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ABSTRACT 


-  We  study-  the  broad  area  of  flow  control  and  sequencing  in  computer  communication 
networks. 

A  special  case  of  message  sequencing  is  shown  to  reduce  to  a  Markov  process.  A  more 
general  sequencing  model  is  also  studied;  only  approximate  results  are  obtained  in  this  case. 

We  formally  show-  , that  an  end-to-end  flow  control  scheme  in  computer  communication 
networks  can  be  modeled  as  a  closed  network  of  queues.  'Wg  also  introduce, and  analyzer^  flow 
control  scheme  based  on  limiting  the  permit  generation  rates.  It  turns  out  that  this  scheme  is 
very  effective.  More  importantly,  this  scheme  involves  essentially  no  computational  complex¬ 
ity.  This  allows  us  to  measure  and  adjust  the  network  performance  with  little  difficulty.  The 
simplicity  also  allows  us  to  incorporate  more  realistic  aspects  of  network  protocols  such  as 
piggy-back  acknowledgement,  time-out  and  gate  functions  into  our  analysis. 

Two  simple  but  effective  distributed  dynamic  flow  control  schemes  are  proposed  and 
analyzed.  It  is  shown  that  with  either  of  these  two  schemes,  a  network  can  respond  well  to  the 
dynamically  changing  environment. 
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CHAPTER  1 
INTRODUCTION 


1.1  The  Necessity  for  Flow  Control 

Resource  scarcity  is  a  common  phenomenon  in  many  aspects  of  our  lives.  We  are 
accustomed,  for  example,  to  finding  that  there  are  fewer  homes  than  families  needing  them, 
fewer  jobs  than  people  of  working  age,  and  desperately  fewer  doctors  than  patients,  etc. 

How  does  this  manifest  itself  in  a  store-and-forward  computer  communication  network? 
A  computer  communication  network  has  only  a  finite  number  of  buffers,  a  limited  transmission 
bandwidth,  a  finite  nodal  processing  capacity,  and  a  limited  number  of  logical  channels.  Thus  if 
there  is  no  restriction  on  the  entrance  rate  of  data  into  the  network,  sooner  or  later  some 
unpleasant  situation  will  occur.  Some  of  these  unpleasant  situations  can  be  the  loss  of  informa¬ 
tion,  performance  degradation  (to  almost  zero  throughput  and  intolerable  delay)  [GEIS  76], 
unfairness  of  service,  or  even  complete  deadlocks  [KLEI  76]. 


Figure  1.1.  A  computer  communication  network. 

For  example,  consider  the  computer  network  shown  in  Figure  1.1,  where  a  set  of 
sources  and  destinations  (host  computers)  communicate  with  each  other  by  sending  messages 
(or  packets)  through  the  store-and-forward  communication  subnetwork.  Suppose  source  A 
sends  messages  to  destination  X  at  a  rate  faster  than  X  can  consume.  A  queue  of  messages  will 
soon  build  up  at  the  destination  switching  node.  If  the  activity  of  source  A  is  not  stopped 
quickly  enough,  then  either  a  large  number  of  buffers  in  the  network  will  be  occupied  by  source 
A’s  messages  (preventing  the  network  from  serving  others),  or  the  destination  switching  node 
must  discard  some  messages. 
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THROUGHPUT 


Figure  1.2.  Typical  throughput  as  a  function  of  offered  load. 

Even  if  the  input  rate  of  the  sources  is  slower  than  the  output  rate  of  the  destinations, 
a  computer  communication  network  still  cannot  function  without  flow  control.  A  typical  perfor¬ 
mance  degradation  of  computer  communication  networks  without  flow  control  is  depicted  in 
Fig.  1.2.  Here  we  see  that  the  throughput  increases  monotonically  with  the  offered  load  until 
some  threshold  is  reached  Beyond  that  level  the  throughput  collapses  quickly  to  essentially 
zero.  The  causes  of  this  collapse  are  multi-fold  and  are  closely  related  to  the  network  protocols 
at  various  levels 

For  instance,  a  virtual  circuit  computer  communication  network  requires  that  a  com¬ 
munication  path  be  established  before  the  communication  can  start.  If  the  path  cannot  be  esta¬ 
blished  then  the  request  is  refused  IPOUZ  76);  note  that  this  part  of  the  protocol  is  itself  a  flow 
control  scheme  Suppose  we  did  not  refuse  the  request,  but  instead  let  the  data  from  the  new 
link  enter  the  network  and  occupy  buffers  at  the  source  node  while  seeking  a  path  with  free 
buffers  available  for  communication  We  would  most  certainly  find  that  when  the  externally 
offered  load  is  high,  congestion  at  the  source  node  will  cause  the  collapse  in  throughput  illus¬ 
trated  in  Figure  1.2.  (This  congestion  occurs  because  the  node's  buffers  are  now  depleted  and 
it  is  difficult  for  data  to  move.)  Even  worse,  we  may  have  a  complete  deadlock  where  no  traffic 
can  move.  Moreover,  simply  the  fluctuation  of  both  arrival  and  service  rates  can  cause  a  large 
number  of  messages  to  wait  for  service  and  to  accumulate  tn  the  network  Indeed,  the  waiting 
time  may  become  so  long  that  the  sources  decide  that  their  messages  have  been  lost  and  in 
order  to  secure  the  communication,  the  sources  would  retransmit  those  presumably  lost  mes¬ 
sages  (this  is  the  so-called  "timeout"  mechanism,  necessary  because  the  communication  chan¬ 
nels  are  usually  error-prone).  These  retransmissions  may  in  turn  cause  the  network  additional 
overload  and  therefore  an  even  longer  message  waiting  time.  etc.  In  fact,  a  source  may  make 
several  unnecessary  retransmissions  before  it  finally  knows  that  it  can  stop  doing  so  and  thus 
the  throughput  is  severely  degraded  since  much  useless  work  is  done  [KAHN  72).  In  any  case, 
the  fall-off  in  throughput  is  caused  by  wasted  resources  that  are  either  allocated  to  repeating 
work  or  that  are  idle  because  of.  say,  congested  receivers  that  are  in  "refusing"  states,  thus  forc¬ 
ing  the  sender  into  a  "waiting"  state. 
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Studying  Fig.  1.2,  we  see  that  one  should  prevent  the  network  from  operating  in  the 
region  (B,C).  On  the  other  hand,  one  should  also  prevent  the  network  from  operating  near  A 
while  it  could  operate  around  B.  This  can  be  done  by  properly  throttling  the  input  rate  of  mes¬ 
sages  into  the  network  or  by  placing  a  limit  on  the  maximum  number  of  messages  that  can 
exist  in  the  network  simultaneously. 


Figure  1.3.  The  effectiveness  of  flow  control  schemes. 


A  simple  measure  to  test  the  effectiveness  of  a  flow  control  scheme  is  discussed  in 
[KLEI  78],  and  is  shown  in  Fig.  1.3  where  we  see  that  an  ideal  flow  control  scheme  will  accept 
the  total  input  traffic  as  long  as  the  offered  traffic  level  is  less  than  the  network  capacity,  say  y0. 
Beyond  that  the  input  traffic  is  cut  at  the  level  yo-  The  closer  to  the  ideal  curve  a  flow  control 
scheme  can  perform,  the  better  is  the  scheme.  Note  that  a  more  accurate  measure  in  compar¬ 
ing  flow  control  schemes  is  one  that  also  lakes  message  delay  into  consideration.  One  such 
example  is  power,  the  ratio  of  throughput  to  delay  versus  offered  load  [KLEI  79).  For  our  pur¬ 
pose  here  we  shall  assume  that  y0  is  set  such  that  the  message  delay  is  below  some  acceptable 
limit  and  therefore  is  not  critical.  That  is,  we  shall  use  "throughput  versus  offered  load"  as  the 
measure  to  compare  flow  control  schemes. 

In  summary,  flow  control  is  a  set  of  mechanisms  whereby  the  flow  of  data  can  be  main¬ 
tained  within  limits  compatible  with  the  amount  of  available  network  resources  and  whereby  a 
single  user  or  a  single  user  group  is  prevented  from  hoarding  the  resources  of  the  network  to 
the  detriment  of  others  (i.e. ,  fairness  of  service)  [RUDI  76). 

1.2  Existing  Methods  of  Flow  Control 

Various  flow  control  schemes  have  been  investigated  and  implemented  in  the  past;  see 
(GERL  80]  and  the  references  therein.  They  can  largely  be  classified  as  schemes  of  local  .flow 
control  or  global  flow  control.  Examples  of  local  flow  control  schemes  are  the  "minimal 
allocation  -  maximal  limit"  scheme  in  the  ARPANET  [MCQU  72],  the  "square  root  scheme" 
[IRLA  78],  the  "sharing  with  minimal  allocation"  scheme  [KAMO  76],  the  GMD  "buffer  class" 
scheme  [RAUB  76],  These  schemes  are  intended  to  prevent  local  store-and- forward  buffer 
congestion  (local  throughput  degradation)  and  direct  (and  indirect)  store-and-forward  deadlock. 
While  possibly  improving  the  local  network  performance,  a  local  scheme  cannot  optimize  the 


global  network  performance  (high  network  utilization,  high  throughput,  low  delay,  fairness  of 
service,  etc.).  As  an  example,  local  flow  control  stops  an  overactive  user's  input  only  when  the 
backpressure  from  the  local  congestion  propagates  upstream  back  to  the  source.  By  that  time, 
however,  this  user  may  have  captured  an  unreasonably  large  portion  of  network  resources,  and 
other  users  may  be  blocked  from  using  the  network.  To  prevent  this  (and  other)  unhappy 
situations,  global  flow  control  schemes  are  often  used. 

At  first  thought,  one  may  think  that  a  global  flow  control  scheme  should  allocate  net¬ 
work  resources  to  users  in  a  way  such  that,  well  before  the  network  capacity  is  reached,  users' 
inputs  are  throttled  so  that  the  network  will  not  be  overloaded.  Indeed,  this  can  be  accom¬ 
plished  either  by  properly  allocating  a  maximal  number  of  permits  or  by  assigning  an  appropri¬ 
ate  window  size  to  each  group  of  users  (the  group  can  be  as  large  as  that  containing  all  users  or 
as  small  as  that  containing  a  single  user).  The  idea  is  to  keep  the  total  number  of  messages 
simultaneously  existing  in  the  network  below  the  network's  capacity;  that  is,  the  network  as  a 
whole  will  not  be  overloaded.  However,  computer  traffic  is  bursty  and  most  of  the  time  only  a 
small  number  of  users  are  active  simultaneously.  Therefore,  the  network  resources  are  often 
overcommitted  (e.g.,  the  window  size  for  each  user  is  set  such  that  the  sum  of  all  windows  for 
all  users  is  much  greater  than  the  network  capacity)  so  that,  on  the  average,  the  network  utili¬ 
zation  is  kept  high  enough  to  be  reasonably  efficient.  When  resources  are  over-allocated,  the 
network  is  apt  to  become  overloaded  from  time  to  time  as  occasionally  a  majority  of  users 
become  active  simultaneously.  This  (infrequent)  overloading  must  be  handled  by  some  other 
mechanism,  such  as  a  dynamic  or  local  flow  control  scheme  mentioned  above. 

In  short,  local  flow  control  is  usually  used  to  resolve  local  congestion  and  deadlocks, 
while  global  flow  control  is  used  to  achieve  high  network  performance  (high  utilization,  high 
overall  throughput,  tolerable  admission  delay,  lower  delivery  delay,  fairness  of  service,  etc.) 
most  of  the  time,  and  further  to  guarantee  that  when  the  network  is  overloaded  it  is  overloaded 
gracefully,  i.e.,  network  resources  are  allocated  fairly  among  users. 

In  this  dissertation  we  concentrate  only  on  global  flow  control.  Existing  methods  of 
global  flow  control  can  be  largely  classified  into  the  three  categories  described  below. 

1.2.1  Isarithmic  Flow  Control 

This  method  was  originally  suggested  by  Davies  (DA VI  72},  and  has  been  implemented 
by  Price  in  his  computer  simulation  model  [PRIC  77).  The  network  is  initially  provided  with  a 
fixed  number  of  "permits”,  several  of  which  are  held  in  store  at  each  node.  As  traffic  is  offered 
by  the  local  host  computer  to  the  subnet,  each  packet  must  secure  a  permit  before  admission  to 
the  subnet  is  allowed.  Each  accepted  packet  temporarily  consumes  (i.e.,  seizes)  a  permit  from 
the  accepting  node's  store.  When  a  packet  reaches  its  destination,  the  permit  is  released  and  is 
either  left  at  the  destination  node  for  its  use  or  shipped  elsewhere  for  the  use  of  others. 

The  main  advantage  of  this  scheme  is  that  it  detects  network  congestion  and  automati¬ 
cally  throttles  input  into  the  network  (the  detection  and  throttling  are  automatically  done  since 
there  are  only  a  fixed  number  of  permits). 

The  difficulty  of  this  scheme  lies  in  finding  a  good  adaptive  way  for  distributing  these 
permits  so  as  to  maximize  throughput  or  minimize  delay.  It  has  been  shown  in  (PRIC  73]  that 
the  original  isarithmic  control  scheme  cannot  by  itself  secure  the  fairness  of  service.  Moreover, 
a  network  usually  deals  with  different  types  of  data.  For  example  (see  [OPDE  74] ) ,  data  may 
be  of  the  HT  (high  throughput)  type  as  in  the  case  of  file  transfer,  the  LD  (low  delay)  type  as 
in  the  case  of  time  sharing,  or  of  the  RT  (real  time)  type  as  in  the  case  of  speech  transmission. 
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Different  types  of  data  require  different  ways  of  distributing  permits.  The  HT  type  prefers  the 
network  to  allocate  a  large  number  of  permits  to  current  active  sources  so  that  the  throughput 
can  be  maximized.  The  LD  type  would  rather  the  network  make  sure  each  source  always  has 
several  permits  at  hand  so  that  the  admission  delay  is  minimized.  The  RT  type  requires  that 
once  a  source  is  allowed  to  transmit  its  messages,  enough  permits  should  be  given  so  that  the 
real-time  nature  of  the  traffic  is  not  distorted.  So  far  there  exists  no  easy  way  to  effectively  dis¬ 
tribute  the  permits. 

1.2.2  End-To-End  Flow  Control 

Typically  most  networks  are  based  on  the  concept  of  logical  channels*  and  each  logical 
channel  is  end-to-end  controlled.  Examples  of  end-to-end  flow  control  are  SNA  pacing  [IBM 
75],  RFNM  in  the  ARPANET  (MCQU  72,  KLEI  76]  and  various  window  mechanisms  [POUZ 
76,  CERF  74].  Important  functions  of  such  end-to-end  flow  control  are  to  synchronize  the 
source  input  rate  to  the  sink  acceptance  rate,  and  to  protect  the  network  from  being  overloaded 
by  a  small  number  of  users.  All  of  these  schemes  work  by  limiting  the  number  of  messages  in 
a  logical  channel.  Suppose  that  L,  is  the  maximum  number  of  messages  in  logical  channel  /, 
and  that  the  network  has  a  total  of  K  logical  channels.  Then  the  maximum  number  of  mes¬ 
sages  permitted  to  enter  the  network  is 

Wmax  -  +  L2  +  '  '  '  +  Lk 

Due  to  the  bursty  nature  of  the  data  sources,  is  usually  set  higher  than  the  net¬ 
work  can  actually  handle,  e  g.,  at  point  C  of  Figure  1.2.  Through  averaging,  the  network  utili¬ 
zation  is  hopefully  at  point  B  of  Figure  1.2.  In  other  words,  .V'™,  is  set  according  to  the  over¬ 
commitment  principle.  Consequently  there  is  a  finite  probability  that  all  sources  would  be 
active  simultaneously,  thereby  reaching  <Vm,x,  and  the  network  would  be  driven  into  the  region 
beyond  point  C  of  Figure  1.2.  Thus,  end-to-end  flow  control  is  not  by  itself  sufficient  to  keep 
the  network  from  being  congested.  We  also  need  a  way  (perhaps  dynamic)  to  limit  the  total 
number  of  messages  in  the  network. 

1.2.3  Hybrid  Flow  Control 

This  scheme  is  essentially  the  combination  of  isarithmic  and  end-to-end  control 
schemes.  The  communication  sources/ destinations  are  divided  into  exclusive  groups  and  each 
group  is  provided  with  a  fixed  number  of  permits.  Above  this,  the  total  number  of  messages 
from  all  groups  is  also  restricted.  An  example  of  this  is  shown  in  [WONG  77],  We  will  discuss 
this  further  in  the  following  section. 

1.3  Existing  Models 

Due  to  the  complex  multi-variate  environment  of  flow  control,  quantitative  and  analytic 
results  have  been  few.  Among  them  are: 


*  A  logical  channel  is  a  conceptual  connection  between  a  source  node  and  a  destination  node 
or  a  source  process  and  a  destination  process. 


1.3.1  Wong  and  Unsoy’s  Two  Level  Flow  Control  Model  IWONG  771 


J.  W.  Wong  and  M.  Unsoy  considered  a  fairly  general  flow  control  scheme  in  which 
flow  is  controlled  at  various  levels.  For  example,  at  the  first  level,  an  upper  limit  L  is  set  for 
the  total  number  of  messages  allowed  into  the  network;  in  the  second  level,  an  upper  limit  Lr 
is  set  for  the  r*  disjoint  group  of  source-destination  pairs.  In  fact,  this  is  the  hybrid  scheme 

mentioned  in  the  last  section.  Observe  the  generality  of  this  model.  If  L  —  where  K  is 

r— 1 

the  total  number  of  logical  channels,  then  it  is  an  end-to-end  flow  control  scheme.  On  the 
other  hand,  if  L,  =  L  for  all  r  =  1,  2,  •••,£,  then  it  is  a  pure  isarithmic  flow  control 
scheme. 


Equipped  with  the  tools  from  the  theory  of  closed  queueing  networks  (GORD  67], 
[BASK  75),  Wong  and  Unsoy  were  able  to  solve  for  the  equilibrium  state  of  a  computer  com¬ 
munication  network  with  any  topology,  any  loop-free,  non-merging  (after  splitting)  routing 
scheme  and  any  traffic  matrix. 

Although  their  work  may  be  considered  as  an  elegant  academic  achievement,  the  model 
implicitly  assumes  that  each  node  can  obtain  information  (such  as  "how  many  messages  are  in 
the  network  now?")  at  no  cost,  and  unfortunately,  this  is  unrealistic.  In  a  real  network,  control 
information  is  either  piggy-backed  or  sent  as  an  ordinary  packet  through  the  network.  Thus  it 
takes  time  for  one  to  gather  information  and  the  information  one  gets  is  partially  "out  of  date". 
In  fact,  this  is  one  of  the  factors  motivating  our  investigation  of  new  models  which  take  this 
delay  in  gathering  information  into  consideration. 

1.3.2  Kleinrock  and  Kermani’s  End-To-End  Flow  Control  Model  IKLE1  80a) 

In  this  model,  a  network  is  viewed  as  a  channel  whose  delay  is  a  function  of  the  offered 
load  (namely,  a  multi-stage  Eriangian  delay  distribution  with  parameter  as  a  function  of  offered 
load).  Under  this  assumption  (and  other  usual  assumptions)  Kleinrock  and  Kermani  were  able 
to  show  the  effects  of  window  size,  time-out  duration,  and  destination  buffer  size  on  network 
performance.  A  dynamic  scheme  based  on  Markovian  decision  processes  [HOW A  60]  that  gave 
a  static  control  policy  based  on  the  number  of  available  empty  buffers  at  the  destination  was 
also  investigated. 

Two  weak  points  of  the  model  are  (1)  it  is  difficult  to  find  the  exact  relationship 
between  the  network  delay  and  the  offered  load  (on  which  the  entire  analysis  is  based),  and  (2) 
the  topology  of  the  network  (which  plays  an  important  role  in  network  performance)  is  ignored 
so  that  the  effect  of  side  traffic  was  not  taken  into  consideration. 

1.3.3  Lam  and  Reiser's  Input  Buffer  Limits  Model  ILAM  771 

Based  on  the  idea  of  GMD’s  buffer-class  model  [GIES  76],  Lam  and  Reiser  modeled  a 
computer  communication  network  as  a  connection  of  nodes  that  were  themselves  modeled  as 
illustrated  in  Figure  1  4a,  which  shows  how  data  input  from  a  channel  to  a  node  is  routed  to 
other  nodes  through  corresponding  channels  (for  details  see  [LAM  77]).  They  further  dis¬ 
tinguished  between  two  classes  of  data  traffic.  Class  1  is  the  transit  traffic,  and  class  2  is  the 
local  input  traffic.  Class  1  traffic  can  occupy  all  buffers  Nr  of  a  node,  while  class  2  traffic  can 
occupy  no  more  than  A/  buffers  of  a  node.  Lam  and  Reiser  were  able  to  show  that  this  model 
predicted  the  congestion-prone  behavior  (see  Figure  1.2)  of  a  network  if  no  control  in  input 
was  imposed.  The  effect  of  various  A//Ar  ratios  was  also  shown  (see  Figure  1.4b). 
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Figure  1.4(a).  Lam's  model  of  a  node. 
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The  disadvantages  of  this  model  are.  (I)  It  does  not  prevent  the  “capture  phenomenon" 
of  a  network,  that  is,  the  super  active  sources  seize  all  the  resources  of  a  network  while  the  less 
active  sources  are  prevented  from  even  entering  their  data  into  the  network;  (2)  it  is  a  compli¬ 
cated  model  that  requires  an  enormous  amount  of  computational  effort.  For  the  latter  reason 
they  assume  a  "homogeneous  network"  meaning  when  one  node  is  congested,  all  nodes  are 
equally  congested,  which  is  unrealistic,  especially  when  the  network  is  large. 

1.4  Organization  and  Results  of  the  Dissertation 

In  the  last  section,  we  saw  that  there  are  only  a  few  available  flow  control  schemes  and 
we  discussed  their  mathematical  models.  Due  to  the  complicated,  multi-variate  environment, 
one  usually  can  only  use  one's  intuition  to  design  a  flow  control  scheme,  then  one  verifies  and 
improves  this  scheme  through  simulation  or,  occasionally  analysis.  If  one  tries  to  set  up  a 
mathematical  model  for  one’s  scheme,  he  usually  finds  that  in  order  to  obtain  a  model  that  is 
analytically  solvable,  he  must  exclude  certain  aspects  of  his  scheme  (such  as  sequencing,  time¬ 
out  mechanism,  etc.)  and  make  some  convenient  (but  sometimes  unrealistic)  assumptions. 

Moreover,  most  tractable  existing  flow  control  schemes  are  static  rather  than  dynamic. 
They  do  not  adjust  themselves  according  to  the  variation  of  a  network's  state  or  environment. 
A  more  effective  flow  control  scheme  should  be  able  to  detect  the  state  of  the  system  and  its 
environment,  and  then  make  some  proper  real-time  decisions  so  that  the  best  system  perfor¬ 
mance  can  be  attained. 

In  our  study,  we  are  interested  in  inventing  and  analyzing  mathematical  flow  control 
models  based  on  more  realistic  assumptions  and  that  incorporate  more  important  aspects  of  real 
flow  control  schemes.  Further,  we  wish  to  derive  some  dynamic  flow  control  models  that  are 
realistic,  yet  simple  enough  to  allow  mathematical  analysis. 

In  Chapter  2,  we  look  into  the  problem  of  message  sequencing.  First  we  present  a  sim¬ 
ple  case  that  allows  us  to  describe  sequencing  by  a  Markovian  chain.  Then  we  investigate  a 
more  general  case  for  which  Markovian  analysis  can  no  longer  be  applied.  It  turns  out  that  the 
theory  in  a  GIG/ 1  queueing  system  provides  a  handle  to  deal  with  this  problem.  Using  this 
tool,  we  find  approximation  results  for  the  throughput  and  delay  of  a  model  network  where 
message  sequencing  is  required  before  messages  can  be  delivered  to  their  destinations.  The 
throughput  degradation  caused  by  sequencing  is  quantitatively  obtained.  We  leave  Chapter  2 
with  the  sequencing  problem  still  not  completely  solved;  however  the  finished  work  is  a  step  on 
the  way  toward  the  complete  analytic  solution  and  gives  a  good  guideline  for  engineering 
design. 


In  Chapters  3  and  4,  we  examine  two  new  static  flow  control  methods.  The  method  in 
Chapter  3  transforms  an  'open'  computer  communication  network  [JACK  63]  (open  in  the 
sense  that  data  flows  in  and  out)  into  a  closed  queueing  network  [GORD  67],  Unfortunately 
because  of  the  complexity  of  computation  involved,  only  a  heuristic  solution  is  possible. 

Two  schemes  in  Chapter  4  create  a  new  dimension  in  flow  control  methods  by  placing  a 
limit  on  the  permit  generation  rates.  These  schemes  assign  a  fixed  rate  for  generating  permits 
(instead  of  using  a  fixed  window  size  for  unacknowledged  permits)  for  the  entire  network  (as  in 
the  first  scheme)  or  for  each  logical  channel  (as  in  the  second  scheme).  It  is  then  shown  that 
by  doing  so  we  obtain  a  flow  control  scheme  that  is  fairly  easy  to  implement.  Later,  the  second 
scheme  is  improved  to  an  ideal  one  (i.e.,  it  can  throttle  the  input  traffic  according  to  the  ideal 
curve  as  shown  in  Figure  1.3).  With  this,  the  mechanisms  of  piggy-back  acknowledgement  and 
time-out  are  then  easily  incorporated  into  the  mathematical  model.  Two  cases  of  piggy-back 
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acknowledgements  are  examined.  The  first  one  is  based  on  the  number  of  ACKs  that  accumu¬ 
late  while  they  wait  to  be  piggy-backed.  Once  the  number  exceeds  a  limit  L,  a  special  control 
message  is  generated  to  piggy-back  all  of  the  waiting  ACKs.  To  avoid  indefinite  waiting  of 
ACKs  when  the  opposite  traffic  is  low,  the  second  case,  which  is  based  both  on  the  number  of 
ACKs  accumulated  and  the  length  of  time  these  ACKs  have  been  waiting,  is  investigated.  In 
the  time-out  scheme,  the  effect  of  channel  error  probability  and  the  length  of  the  time-out 
period  are  presented.  Also  in  Chapter  4,  we  move  into  the  area  of  inter-netting.  In  this  area 
we  show  that  our  proposed  flow  control  scheme  can  be  nicely  incorporated  into  the  protocol 
that  governs  the  flow  control  function  of  a  gateway  through  which  data  of  different  networks 
pass. 


In  Chapter  5,  we  then  look  into  dynamic  flow  control  schemes.  There  we  propose  and 
analyze  two  simple  distributed  dynamic  flow  control  methods.  In  these  methods,  no  central 
node  is  needed  to  oversee  the  state  of  the  entire  network.  The  sources  themselves,  cooperating 
with  the  physical  channels  through  which  the  sources’  data  pass,  can  control  the  flow  dynami¬ 
cally  and  distributedly,  based  on  measurements  or  information  exchanges.  To  avoid  the 
extreme  complexity  involved  in  the  exact  analysis,  only  a  semi-quantitative  analysis  is  given, 
and  only  simulated  results  are  presented.  These  distributed  dynamic  schemes  adapt  to  the 
changing  environment  effectively.  In  particular,  throughput,  delay  and  permit  generation  rate 
allocation  versus  time-varying  external  applied  loads  are  presented. 

We  consider  our  studies  in  this  dissertation  as  contributing  to  the  important  and 
neglected  frontier  of  modelling  and  analyzing  the  protocols  of  distributed  computer  communica¬ 
tion  networks. 

The  results  we  obtain  in  Chapter  2  shed  light  on  the  way  to  solve  the  almost  hopeless 
analytic  problem  of  message  sequencing,  where  the  different  message  positions  represent 
different  states  of  a  network  (in  addition  to  the  number  of  messages  existing  in  the  network). 
The  observation  made  in  Chapter  3  enables  us  to  describe  a  computer  communication  network 
as  a  dosed  network  of  queues.  This  greatly  simplifies  the  network  modeling,  at  least  conceptu¬ 
ally  (regardless  of  the  computational  complexity).  In  Chapter  4,  we  reveal  a  new  dimension  of 
flow  control  techniques,  namely,  those  based  on  limiting  the  permit  generation  rates.  They  are 
as  effective  as  (if  not  better  than)  other  existing  flow  control  schemes.  Yet  they  are  easier  to 
implement,  and  it  is  easier  to  incorporate  other  network  protocols  such  as  piggy-back  ack¬ 
nowledgements,  time-outs,  etc.  into  the  models.  More  importantly,  these  models  are  much 
easier  to  calculate  quantitatively.  No  computational  complexity  is  involved  at  all  —  a  strong 
contrast  to  other  existing  schemes  and  models.  The  dynamic  flow  control  schemes  investigated 
in  Chapter  5  provide  simple  dynamic  distributed  flow  control  methods  that  are  a  must  for 
today's  networks  with  their  rapidly  expanding  size,  arbitrary  topology,  and  heterogeneous  traffic 
matrices. 

Our  studies  provide  a  tool  to  control  the  data  flow  of  large,  complicated  computer  com¬ 
munication  networks  and  to  describe,  in  a  simple  manner,  the  effects  of  flow  control  schemes 
on  these  networks. 


9 


CHAPTER  2 

THE  SEQUENCING  PROBLEM 

The  models  discussed  in  Chapter  !  do  not  consider  the  important  area  of  message 
sequencing.  Usually,  a  data  terminating  device  (e.g.,  a  host  computer)  requires  messages  be  in 
the  correct  sequence  before  they  can  be  accepted  for  delivery.  If  the  network  takes  the  respon¬ 
sibility  for  sequencing,  messages  occupy  the  buffers  of  the  destination  switching  node  longer 
than  when  sequencing  is  not  required,  and  therefore  throughput  is  reduced  while  the  delay  is 
increased.  This  has  been  partially  investigated  in  [SUNS  75)  but  no  significant  result  was 
obtained.  In  the  following  section  we  present  and  analyze  two  models  in  order  to  appreciate  the 
complexity  of  the  sequencing  problem  and  to  investigate  its  effect  on  the  network  performance. 

As  messages  pass  through  the  network  they  may  take  different  paths,  or  they  may  be 
corrupted  by  noise  (in  which  case  they  will  be  retransmitted)  so  that  they  may  arrive  at  the  des¬ 
tination  node  out  of  sequence.  What  if  the  destination  node  refuses  to  accommodate  the  out- 
of-sequence  messages,  or  what  if  it  accommodates  them  but  won't  deliver  them  until  they  are 
in  the  correct  sequence?  The  exact  analysis  of  such  a  problem  is  difficult  because  the  state 
description  now  involves  not  only  the  number  of  messages  in  various  queues  but  also  their 
relative  positions  in  these  queues.  As  a  network  grows  large,  the  number  of  states  also 
becomes  explosively  large  and  soon  diminishes  our  capability  to  solve,  or  even  describe  the  sys¬ 
tem. 


2.1  A  Simple  Example 

To  appreciate  the  complexity  of  the  problem,  let  us  first  consider  a  system  as  shown  in 

Figure  2.1,  where  we  have  a  loss  system  with  a  Poisson  message  arrival  rate  k.  Let  us  assume 

that 

(Al)  A  window  size  W  (represented  by  the  number  of  tokens)  is  imposed  to  limit  the  input 
traffic. 

(A2)  The  round-trip  delay  time  (message  plus  ACK  delay)  is  exponential  with  rate  q.. 

(A3)  There  is  a  buffer  of  size  B  and  an  output  channel  with  infinite  speed  at  the  destination 

node. 

(A4)  The  buffer  B  accommodates  the  out-of-sequence  messages,  but  when  a  message  (say 
message  a)  arrives  to  find  the  buffer  B  full,  if  any  messages  in  B  are  more  out-of¬ 
sequence  than  message  a ,  the  buffer  will  eject  (kick  out)  the  most  out-of-sequence 
message  among  those  in  the  buffer  in  favor  of  message  a. 

(A5)  The  output  channel  will  emit  messages  with  infinite  speed  once  they  are  in  sequence. 

(A6)  When  the  buffer  receives  a  message,  it  returns  a  positive  ACK;  when  it  rejects  a  mes¬ 
sage  because  there  is  no  vacancy,  it  returns  a  negative  ACK. 
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Figure  2.1.  An  illustrative  network  model  for  the  sequencing  problem. 


(A7)  The  source  immediately  retransmits  the  rejected  message  once  it  receives  the  negative 
ACK. 

(A8)  When  a  message  is  kicked  out  of  the  buffer  to  make  room  for  a  less  out-of-sequence 
message,  this  information  is  piggy-backed  in  the  positive  ACK  for  the  accepted  message 
so  that  the  source  will  transmit  the  kicked  out  one  again. 

(A9)  The  channels  are  noiseless,  and  the  network  is  reliable. 

To  illustrate  how  we  can  mathematically  describe  and  solve  the  system,  let  us  consider 
a  simple  case  in  which  the  window  size  W  =  4,  and  the  buffer  size  B  —  2. 

From  the  viewpoint  of  the  source,  we  can  describe  the  state  of  the  system  with  a  state 
vector,  for  example  (1,  1,  1*,  D,  meaning  that  the  source  has  sent  four  messages.  Among 
them  are  the  positive  ACKs  for  the  third  and  fourth  messages  (we  number  from  left  to  right) 
that  have  come  back,  and  since  they  are  out  of  sequence,  we  denote  the  third  and  the  fourth 
positions  of  the  state  vector  by  two  "l*"s.  These  two  messages  are  fully  occupying  the  destina¬ 
tion  buffer  (B  =  2).  The  simple  "1"  indicates  that  a  message  has  been  sent  but  its  ACK  is  not 
back  yet.  Note  that  if  now  the  +ACK  for  the  second  message  comes  back  (which  also  informs 
the  source  that  the  fourth  message  has  been  kicked  out),  the  state  vector  will  become  (1,  1*, 
1*,  1)  since  the  source  knows  now  that  the  fourth  message  has  been  kicked  out  (by  A4)  to 
make  room  for  the  second  message,  the  source  retransmits  the  fourth  immediately  (by  A7).  If 
now  the  +  ACK  for  the  first  message  comes  back,  the  source  will  change  the  state  vector  to  (1, 
0,  0,  0)  since  it  knows  that  the  first,  second,  and  third  message  are  now  in  sequence  and  are 
zipped  out  already  (by  A5),  and  the  fourth  message  now  becomes  the  first  one.  Zero  here 
denotes  "no  activity".  Thus  the  transition  diagram  of  the  states  can  be  shown  as  in  Figure  2.2. 
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Figure  2.2.  The  state  transition  diagram  from  source’s  viewpoint. 


We  see  that  there  are  1 5  states  for  this  simple  example.  When  W  and  B  are  large,  we 
are  in  trouble.  In  fact,  the  number  of  states  is  2W  (when  B  >  W).  Another  difficulty  is  that 
we  are  unable  to  draw  a  state  transition  diagram  (no  matter  how  complicated)  for  a  system  with 
general  message  arrival  and  network  delay  processes.  This  case  will  be  discussed  in  the  next 
section. 


In  Figure  2.3  we  show  the  analytically  calculated  throughput  and  delay  of  a  system  with 
W  «  8  and  B  *>  2,4,6,8.  The  solid  curves  are  for  throughput,  and  the  dashed  curves  are  for 
delay.  Note  that  as  m/  X  becomes  large,  (i.e.,  the  network  delay  becomes  shorter  than  the  mes¬ 
sage  interarrival  time),  the  throughput  approaches  X  and  is  not  affected  by  the  size  of  the  buffer 
since  the  chance  for  messages  to  get  out  of  order  is  small.  In  general,  the  smaller  the  buffer 
size  and  f*/x,  the  smaller  is  the  throughput  and  the  larger  is  the  delay  since  a  lot  of  kicking  out 
occurs  at  the  buffer. 

2.2  A  More  General  Case 

Now  let  us  consider  a  more  general  case.  The  system  is  similar  to  that  in  Figure  2.1 
except  that  the  the  network  delay  is  generalized  to  an  arbitrary  one-way  delay,  X  (X  is  a  ran¬ 
dom  variable),  and  the  service  time  at  the  output  channel  is  arbitrary  and  is  described  by  a  ran¬ 
dom  variable  Y  with  some  distribution. 
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•e  2.3.  Throughput  and  delay  curves  for  the  system  in  Fig.  2.1. 


No  state  diagram  similar  to  that  in  Figure  2.2  can  be  drawn  here.  To  exactly  solve  this 
generalized  system  is  almost  hopeless.  It  is  more  difficult  than  G/G/l  IKLEI  75]  which  itself  is 
only  solvable  using  very  complicated  means.  Hence  only  an  approximation  (namely,  curve 
fitting)  is  given  here.  We  first  look  into  the  case  B  —  °o  and  then  we  turn  to  the  case,  B— 0. 
Moreover,  we  know  that  if  there  is  no  output  channel  the  throughput  is  zero.  We  thus  obtain 
three  points  for  three  different  cases  and  then  we  fit  a  curve  exponentially  through  these  points. 

Case  I: 
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Figure  2.4.  The  time  diagram  when  B=«®. 

The  system  behavior  can  be  described  by  the  time  diagram  in  Figure  2.4.  Here,  is 
the  random  variable  representing  the  waiting  time  of  the  k'h  customer  (accepted  to  the  net¬ 
work)  in  the  destination  buffer  of  infinite  size,  and  tJ+l  represents  the  ume  duration  between 
the  j'h  and  the  j  +  \'h  accepted  customers.  (Although  we  use  the  "IF"  to  represent  the  token 
number,  window  size,  and  waiting  time,  the  reader  should  not  be  confused  as  the  meaning  is 
clear  from  the  context.) 

To  understand  the  time  diagram  shown  in  Figure  2.4,  let  us  follow  the  accepted  custo¬ 
mer  C„.  As  he  arrives  at  the  entrance  of  the  network,  he  secures  a  permit  and  is  allowed  to 
enter  the  network,  where  he  is  transmitted  toward  his  destination.  It  takes  him  X„  units  of 
time  to  arrive  at  the  destination.  He  then  joins  the  buffer  B  where  he  waits  for  Wn  units  of 
time  to  get  in  the  right  order,  or  for  his  turn  to  be  delivered.  It  then  takes  the  output  channel 
another  Y„  units  of  time  to  deliver  him.  C„+1  may  arrive  at  the  buffer  before  C„  (as  shown  in 
the  figure).  Thus,  we  have 

K+t  +  *n+1  =  [W„+Xn+Y-tn+^X'"  (2.1) 


where  we  define 


By  letting  W'  =  W+X  we  obtain 


[A)+B 


if  A>B 
\iA^B 


WW  =  l*V  +  Yn~<n+ x\*X'*1 

We  assume  that  message  r?+l  is  an  "average  message"  so  that  Xn+\  has  the  value  X.  Then  the 
above  equation  becomes 

"W  =  l  +  VnV* 

=  max[Y,  W„'+Un]  (2.2) 

where  U„  4  Y„-i„+\.  Observe  that  rn+),  the  interarrival  time  between  accepted  messages  n+ 1 
and  w,  is  not  independent  of  X„  (hence  1+y).  This  can  be  seen  at  the  extreme  case  where  we 
have  only  one  token  for  the  entire  system  so  that  if  X„  is  large  tn+\  is  also  large.  Yet  the 
dependence  becomes  less  and  less  prominent  as  W,  the  number  of  tokens,  becomes  larger  and 
larger.  Therefore,  we  assume  that  r„+i  is  independent  of  W„'  (i.e.,  1V>  >  1) .  This  assumption 
implies  the  independence  of  W„'  and  U„.  Thus,  a  method  similar  to  Kingman's  Algebra  for 
G/G/l  [KING  661  can  be  used  to  solve  (2.2).  That  is,  to  find  w„+i(w),  the  pdf  of  W^+i,  we 
calculate  the  pdf  of  Wn  +  Un  by  convolution,  and  sweep  the  total  probability  with  arguments 
<X  up  to  X.  The  iteration  continues  until  it  converges  [KLE1  75], 


To  calculate  the  throughput,  S,  suppose  that  we  have  W  tokens  for  the  source,  and 
that  these  tokens  travel  between  the  source  and  the  destination  with  a  round  trip  delay  equal  to 
DX  =  X  +  X.  Then  [GROS  74l  the  system  behaves  like  an  M/G/W  system  with  loss;  thus, 

P,  =  Pr[i  tokens  are  occupied  by  messages) 


=  (KDX)'/i\ 

£  U\DY) '/./!) 

j-0 

Therefore. 

5  =  \(1-PH) 

and  the  average  message  delay  is 

D  =  X  +  W  +  F 


(2.3) 


(2.4) 

(2.5) 


Observe  that  although  the  external  message  arrival  process  is  Poisson,  the  accepted 
arrival  process  is  not;  yet  it  becomes  Poisson  when  W  approaches  infinity. 

With  Equations  (2.2) ,  (2.3),  (2.4)  and  (2.5)  the  system  is  solved  for  the  case  B  — 

Case  2:  B  =  0 

We  now  have  no  buffer  to  store  messages.  A  message  is  rejected  when  it  arrives  at  the  destina¬ 
tion  and  finds  that  the  output  channel  is  busy.  The  "-ACK"  goes  back  to  the  source  (with  delay 
X);  the  source  retransmits  the  message  and  it  may  get  rejected  again.  Thus,  the  message  can 
be  viewed  as  if  it  is  "waiting  in  the  network”.  When  the  output  channel  becomes  free,  the  next 
copy  of  the  message  is  somewhere  in  the  network,  and  is  assumed  to  arrive  at  the  output  chan¬ 
nel  after  DX  (  "  * "  denotes  the  residual  life  of  a  random  variable) .  This  time  diagram  is  shown 
in  Figure  2.5  Now,  with  subscript  n  denoting  the  n'h  message  entering  the  network,  we  have 
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Figure  2.5.  The  time  diagram  when  B 


where  again. 


+  X, 


n  +  l 


\Wn  +  ^  W'-DX'  +  Y„  +  - 


UP* 


-4,  if  /4>fl 
fl,  if 


and. 


A 


if  »F„  >  0 


if  W„  ^  0 


again 


in  define  W'  k  W+X\  then 


,+r 


K+i  -  W  +  *  *.  ■  DX,  +  Yn  -  /,+,]  •*+' 

=  IH'.'  +  A.  •  D>„  +  l/„P* 


=  max  IX,  W'  +  i,  ■  DX„  +  £/„l 


We  see  that  A  *  ■  DX„  represents  the  fact  that  DX„  is  needed  only  if  the  n'h  message  is 
rejected  at  the  destination  when  it  arrives  the  first  time  (i.e.,  M/„>0  ). 

Even  with  the  assumption  that  r„+I  is  independent  of  W„\  Eq.  (2.6)  is  stilly  not  solvable 
by  the  method  described  for  the  B  =  °°  case  because  of  the  existence  of  A  wDXn,  which  is 
again  not  independent  of  Wa'.  For  this,  we  assume  that  A  w  =  1.  That  is,  we  have  a  heavy 
traffic  system  (with  zero  buffers)  so  that  when  a  message  arrives  at  the  destination  the  first  lime 
it  almost  always  finds  that  the  server  is  busy;  and/or  the  distributions  of  X  and  t  are  such  that 
messages  easily  get  out  of  sequence.  Then  (2.6)  becomes 

W„+1'  =  max  [X ,  W +  DX„  +  U„)  (2.7) 

We  further  assume  that  W„'  and  DX„  are  independent  of  each  other.  This  is  not  a  bad  assump¬ 
tion  since  the  server  becomes  free  at  a  random  instant,  and  the  number  of 
reject/ retransmissions  of  message  n  (which  largely  contributes  to  the  value  of  W/„)  is  also  arbi¬ 
trary.  Eq.  (2.7)  then  becomes 

fF„+1'  =  max  [A\fF„'+  U„']  (2.8) 

where 

Un'=Un  +  DXn 

Eq.  (2.8 )  again  can  be  solved  by  iteration.  To  calculate  the  average  throughput  we  first  observe 
that  the  average  time  period  D  that  a  token  is  occupied  by  a  message  is 

D-X  +  W  +  DX  +  X  (2.9) 


where 

W  =  W'  -  X 


Thus, 


P»  =  Pr [all  W  tokens  are  occupied  by  messages] 


=  [(AO)"  /  W!]/[£(AD)V./!l 

7-0 

(2.10) 

Therefore,  the  average  throughput  is 

S  =  X 1 1  —  Pw\ 

(2.11) 

and  the  message  delay  is 

T=X+W  +  DX+Y 

(2.12) 

With  Equations  (2.8),  (2.9),  (2.10),  (2.11),  and  (2.12)  we  solve  the  system  for  case 
B  =  0.  under  the  assumptions  described  above. 

In  Figure  2.6,  we  show  the  calculated  curves  and  the  simulated  results  for  the  case 
X — and  )—  yeyy  with  m  “  0.4,  y  =  1.2  and  \  -  1.0.  Note  that  the  point  S+B- 0  on 
the  horizontal  axis  corresponds  to  the  case  where  we  have  no  server;  otherwise,  S— 1. 

In  Figure  2  7  we  show  the  comparison  between  two  systems  where  one  requires 
sequencing  and  the  other  does  not.  In  this  figure,  as  fl  —  °o  the  throughput  of  both  schemes 
matches  nicely  except  when  the  window  W  is  small.  This  discrepancy  at  small  W  occurs 
because  the  assumption  that  the  duration  between  two  successive  acceptances  of  messages  is 
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Figure  2.6.  The  calculated  and  simulated  results  for  generalized  sequencing  problems, 


exponential  and  is  independent  of  one  way  message  delay  is  no  longer  valid  when  W  is  small. 
We  know  that  when  both  W  and  8  are  large  the  exponential  and  independence  assumptions 
become  correct  and  the  rejection  activity  also  becomes  low.  Therefore  sequencing  causes  little 
reduction  in  throughput  (yet  it  does  cause  an  increase  in  delay  because  of  the  time  required  to 
reassume  order). 

The  careful  reader  should  have  noticed  that  in  the  curve  fitting  approximation 
presented  above,  the  "kicking  out"  mechanism  (see  (A4)  and  (A8)  in  Sec.  2.1)  does  not  have  a 
chance  to  play  its  role.  We  have  only  obtained  approximated  results  for  8  =  <»,  0  with  5  =  1, 
and  for  8=0  with  5  =  0  also.  Yet  for  the  "kicking  out"  mechanism  to  be  active  we  need 
8  >  0,  finite,  and  5  >  0 

Despite  the  weakness  of  our  analysis,  we  believe  that  the  approach  we  take  here  is 
valuable.  Recently,  in  [KAMO  81]  and  [BARB  81]  fruitful  results  have  been  obtained. 
Interested  readers  should  refer  to  the  above  references. 


Figure  2.8.  Message  sequencing  nodes. 


Another  interesting  question  is  as  follows:  Consider  the  system  in  Figure  2.8.  Assume 
that  all  processes  are  Poisson  (or  exponential).  Messages  enter  the  network  with  rate  A.  Then 
they  arrive  at  the  destination  node  after  an  exponential  delay  at  rate  /u.  The  destination  node 
then  delivers  (at  rate  y)  the  messages  to  the  user  whose  consumption  rate  is  6.  The  question 
to  ask  is  which  scheme  is  better  -  to  carry  out  message  sequencing  at  the  destination  node  or  at 
the  user  node. 

Intuitively,  the  better  place  to  resume  order  is  at  the  node  where  the  service  rate  is 
slower.  This  is  because  a  message  usually  must  wait  here  anyway  to  be  served  later  and  there¬ 
fore  there  is  little  extra  cost  involved  while  waiting  to  resume  the  correct  sequence.  Yet  the 
analytic  proof  is  difficult  because  the  departure  process  from  the  destination  switching  node  is 
difficult  to  find.  The  interested  reader  may  wish  to  pursue  this. 


20 


CHAPTER  3 

STATIC  FLOW  CONTROL  BASED  ON  ASSIGNING 
A  FIXED  NUMBER  OF  PERMITS  PER  LOGICAL  CHANNEL 


In  Section  1.2.1,  we  commented  on  a  limitation  of  the  Wong-Unsoy  two-level  control 
model,  namely,  it  did  not  take  into  account  the  time  needed  to  gather  control  information  (e.g., 
the  number  of  messages  now  in  the  network). 

In  this  chapter  we  try  to  overcome  this  limitation  by  considering  a  new  end-to-end  flow 
control  model.  In  this  model,  each  logical  channel  (a  source-destination  pair)  is  provided  with 
a  fixed  number  of  permits.  A  message  must  secure  a  permit  at  its  originating  node  before  it 
can  enter  the  network.  After  the  message  reaches  its  destination,  the  accompanying  permit  is 
then  carried  back  to  the  original  node  by  the  acknowledgement  of  that  message.  Thus,  a  source 
knows  that  a  message  has  left  the  network  only  when  it  receives  the  ACK  for  that  message. 

3.1  System  Assumptions 

Consider  a  store-and-forward  computer  communication  packet-switching  network  with 
any  topology  and  any  input  traffic.  We  make  the  following  assumptions. 

(Al)  Each  node  has  an  infinite  number  of  buffers. 

(A2)  The  external  traffic  arrival  rate  for  the  /•"'  logical  channel  is  Poisson  with  a  rate 

M  ,-<'■> 

(A3)  Each  logical  channel  r  is  allowed  to  accommodate  no  more  than  L,  unack¬ 
nowledged  messages  (i.e..  logical  channel  r  has  Lt  permits). 

(A4)  We  have  exponential  service  time  for  messages  and  the  same  exponential  ser¬ 
vice  time  for  acknowledgements. 

(A5)  We  have  a  fixed  routing  scheme. 

(A6)  Kleinrock’s  independence  assumption  [KLE1  761  holds. 

3.2  An  Interesting  Observation 

The  following  observation  enables  us  to  transform  an  open  network  of  queues  (for 
messages)  into  a  closed  network  of  queues  (for  permits).  Consider  a  queue  of  permits  as 
shown  in  Fig.  3.1.  Let  us  assume  that  the  permit-arrival  process  Gp  is  an  arbitrary  one.  The 
permits  are  queued,  the  permit  at  the  head  of  the  queue  can  leave  the  queue  only  when  a  mes¬ 
sage  comes  and  (immediately)  takes  it  out  (when  an  arriving  message  finds  no  permit  it  will 
just  turn  away  and  never  come  back).  If  the  arrival  process  of  messages  is  Poisson,  then  we 
can  model  the  queue  for  permits  as  a  Gn/  M/\  queueing  system  as  in  the  lower  half  of  Fig.  3.1. 


PERMITS 


Mio 


MESSAGES 
(Poisson  process) 


EQUIVALENT 


SERVER 

PERMITS  (exponential) 


Figure  3.1.  Two  mathematically  equivalent  systems. 


3.3  The  Modeling  of  the  Flow  Control  Scheme 

With  the  above  observation  and  assumptions  we  are  able  to  model  the  flow  control 
scheme  as  a  closed  network  of  queues  iCiORD  67)  tor  permits.  The  idea  is  best  illustrated  by  the 
following  simple  example.  Consider  two  connected  nodes  communicating  with  each  other  as 
shown  in  Figure  3.2a,  where  /x, .</>,  /  =  1.2.  is  the  external  message  arrival  rate  at  node  i.  The 
queueing  network  in  Figure  3.2b  is  the  model  of  the  How  control  scheme. 

When  a  message  arrives  at  its  source  node  (say  node  1).  it  must  secure  a  permit  (if  no 
permit  is  available,  it  is  rejected  and  never  returns).  Alter  it  secures  a  permit  it  then  enters  a 
queue  (queue  C  in  Fig.  3.2b)  waiting  to  be  transmitted  over  channel  1.  An  accepted  message  is 
finally  sent  to  its  destination  (node  2),  where  the  accompanying  permit  immediately  becomes 
free,  and  is  sent  back  with  the  acknowledgement  to  node  1  (through  queue  D  into  queue  A). 
Thus  we  have  a  closed  network  of  queues  for  the  permits  (fixed  in  number),  and  an  open  net¬ 
work  of  queues  for  messages. 

This  model  is  more  realistic  since  the  time  that  permits  take  to  return  to  the  sources  is 
taken  into  account  (remember  that  Wong-Unsoy  model  mentioned  in  Chapter  1  does  not  incor¬ 
porate  this  in  the  analysis). 

In  Figure  3.2b  we  have  two  closed  chains  of  permits.  One  is  chain  ACDA.  the  other  is 
BDCB.  Chain  ACDA  is  for  permits  used  by  node  1  to  node  2  messages.  Chain  BDCB  is  for 
permits  used  by  node  2  to  node  1  messages.  We  immediately  see  that  the  message  throughput 
(from  node  I  to  node  2)  is 

(jl,A\)  Pr  [queue  A  not  empty  of  permits]  (3.1) 

and  the  one-way  delay  for  accepted  messages  is  the  system  time  in  queue  C.  As  described  in 
[BASK  751,  we  can  denote  a  state  of  the  system  as 
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Figure  3.2.  Illustrative  example  for  the  flow  control  scheme. 


•S’  =  1^1  .  /-! 


with 


A  =  <'s  'x,-  •  's  > 

1  *  K 

K  =  f .  B ,  (  \  /),  and 
V*  =  I  or  2 

depending  on  whether  the  permit  belongs  to  logical  channel  1  (node  1  to  node  2).  or  logical 
channel  2  (node  2  to  node  I  h  and  its  position  in  Z*  denotes  its  actual  position  in  queue  K. 


Thus.  S  can  be  written  as 


S  = 


Vi, 


V, 


Vt 


V, 


The  server  at  queue  K  is  serving  \*  .  the  head  of  the  queue.  Let  L\  =  L2  *=  2  for  simplicity. 
Then  one  of  our  balance  equations  becomes 
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(3.2) 


If  we  let  >.,(1).  t«(2)  and  i<  (/),  »/></).  i  =  I.  2  be  the  solution  of  the  following  four  equa¬ 
tions: 


MU'.)  <  I  >  =  /U4I  /,  ( ) )  =  M  ( '  * 


and 


/y < 2 )  =Mi‘<  =  M4 1/>(2) 

then  it  is  easy  to  see  that  the  solution  to  Kquation  (3.2)  is 


and 


(i  ,v(  (lh„(2)‘V,)(l)  for  /' 


(,  t,  (l)-\,  (2h,,(2)  for  P 


(3.3) 

(3.4) 


In  general  we  have  R  logical  channels  and  N  transmission  channels,  and  each  logical 
channel  r  has  L,  permits.  The  slate  is 


V  =  I/,. 


Al 


where  /  = 


,  with  ,v„  =  1.2. 


R 


The  general  balance  equation  is 

Z'[/,,  .  •••./,.  ■  .  A!  i£  ■  (//*>! 

k  -  I 

=  11  /’I/,.  •/(-).  •/,(+'  *  ./nI  fi(u,>M, <6,  ('..'*  (3  5) 

-I  /-I 


where 


ft (//  ) 


I .  if  >  0 
0.  otherwise 
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Z,(-)  A 


x„ 

,  z,(+xn)  A 

X, 

*  _  J 

x, 

'n 

and 

1,  if  messages  of  logical  channel  r  are  routed 
from  physical  channel  j  to  physical  channel  / 

<t>Jr)  = 

0.  otherwise 

Let  »■,(/•),  /  =  1 .  2,  •  •  ,  V.  r  =  I,  2,  ■  •  •  ,  R  be  the  solution  of  the  following  equations: 

fx,}', (r)  =  (3.6) 

/-I 

Then  the  solution  of  Eq.  (3.5)  is 

P(Zh  •  •  •  .  Zv)  =  G  J]  II  (3.7) 

.-i  *-i 

where  G  is  a  normalization  constant  that  must  be  adjusted  to  force  the  probabilities  to  sum  to 
unity. 

Thus  the  system  is  completely  solved  if  the  value  of  G  is  calculated. 

The  above  flow  control  model,  as  innocent  as  it  may  look,  suffers  from  great  computa¬ 
tional  difficulty  in  determining  the  value  of  key  parameter  G,  the  normalization  constant.  This 
is  the  usual  problem  with  closed  queueing  networks. 

For  an  open  network  of  queues,  G  has  a  simple  closed  form  expression.  Yet  for  a 
closed  network  of  queues,  G  must  be  obtained  by  summing  all  the  feasible  states.  Because  of 
the  combinatorially  growing  slate  space,  a  naive  summation  is  out  of  the  question  except  for 
trivially  small  networks.  Although  some  efficient  algorithms  exist  in  [BUZE  73],  (REIS  75], 
IREIS  76],  [REIS  77]  and  [REIS  78],  the  computational  complexity  is  still  too  much  to  be  prac¬ 
tical. 

For  example,  in  [REIS  76],  the  normalization  constant  is  given  by 
G  =  Prtpopulation  in  the  corresponding  open  network  =  K\ 

—  PrU,  +  ki+  ■  ■  •  +  k\  —  K] 

=  PrU  ,]  ‘PrU2l  *  •  •  •  *  PrU  v] 

where  k,  is  the  queue  size  of  q,,  Pr[k]  is  the  queue  size  probability  of  queue  i.  and  the  aster¬ 
isk  denotes  convolution. 

One  sees  that  these  convolutions  are  all  complex  and  moreover  the  intermediate  results 
may  easily  exceed  the  floating-point  range  of  most  computers  even  though  the  final  results  are 
all  reasonable  in  magnitude.  Had  Wong  applied  his  model  to  a  network  with  a  few  more  nodes 
than  the  simple  network  in  his  paper  [WONG761,  he  would  have  found  himself  confronted  by  a 
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solid  wall  of  computational  complexity. 

This  difficulty  had  made  such  models  infeasible  until  Martin  Reiser’s  breakthrough  in 
which  he  invented  a  heuristic  method  by  using  a  mean  value  analysis  (REIS  78],  We  describe 
this  method  in  the  rest  of  this  chapter. 

3.4  The  Mean  Value  Analysis  and  the  Heuristic  Solution 

3.4.1  Mean  Value  Analysis  in  a  Closed  Cyclic  System 

The  ideas  behind  the  mean  value  analysis  are  easiest  to  describe  by  using  the  example 
of  a  dosed  cyclic  queue.  We  denote  by  -S’ (A)  the  queueing  system  with  A  customers.  Furth¬ 
ermore.  denote 

A'.  Number  of  customers  in  the  system 

A':  Number  of  queues  in  the  cyclic  chain 

r,:  Mean  service  time  of  queue  i 

t,:  Mean  system  time  of  queue  /  (including  service) 

//, :  Mean  queue  size  of  queue  t  (including  customer  in  service) 

X:  Throughput  of  the  chain  (=  circulation  rate) 

We  will  use  parenthetical  arguments  if  we  wish  to  emphasize  that  quantities  are  for  the  system 
with  population  size  A,  viz  //,(A),  t,(K)  and  A ( A' )  are  the  mean  queue  size,  the  mean  system 
time  and  the  throughput  of  S(K).  We  can  straightforwardly  write  the  equations 

i,(K)  =  r,  +r,  •  (mean  number  of  customers  seen  upon  arrival]  (3.8) 

A  ( A'  >  =  ^ -  (3  9) 

I  /.(AT) 

-i 

//,  ( A )  =  A  ( A )  r,  ( A )  (3.10) 

Equation  (3.8)  states  that  an  arriving  customer,  on  the  averaee.  must  wait  for  its  own  service 
time  plus  the  backlog  of  work  seen  upon  arrival  at  queue  /.  Equations  (3.9)  and  (3.10)  are 
simply  Little’s  formula  applied  to  the  entire  chain  and  to  each  queue  respectively.  Note  that 
the  average  number  of  customers  is  of  course  A.  If  we  had  an  expression  for  the  bracketed 
term  in  Eq.  (3.8)  we  could  solve  for  all  unknown  quantities.  Such  an  expression  is  found  using 

Theorem  1:  In  a  dosed  queueing  network  with  product-form  solution,  the  probability  that  an  arriving 
customer  finds  state  k  in  the  system  S(K)  is  the  same  as  the  long  term  equilibrium  probability  of  k  in 
S(K-I). 


From  the  theorem  immediately  follows  the  mean  waiting  time  equation 

r,(A)  -  t,  +  t,«,(A-1)  (3.11) 


Equation  (3.11)  together  with  (3.9)  and  (3.10)  can  be  solved  easily  in  a  recursive  manner. 
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//  (0)  =  0 


(3.12) 


/  < A.  )  =  T,|l+//  < A  —  1 )  1 

(3.13) 

X  4  At  )  =  .  X  >  0.  »  =  1.  2,  •  .  ‘ 

2. ,  ( .  < X  ) 

V  (3.14) 

,1  (A  )  =  X  ( At  )  /.  ( At  ) 

(3.15) 

3.4.2  Generalization  to  Networks  with  Many  C  losed  Chains 

The  recursion  (3  12)  to  (3  15)  generalizes  easily  to  a  network  with  a  full  routing  matrix 
Let  /'  denote  an  arbitrarily  chosen  queue.  Then,  the  quantity  is  defined  by 

n  ■  =  1 

(3.16) 

«  =  HP 

(3.17) 

where 

«  =  ) 

measures  the  average  number  of  visits  a  customer  makes  to  queue  /  between  successive  visits 
to  the  marked  queue  i‘.  Since  the  number  of  visits  and  the  throughput  are  proportional,  fl, 
also  measures  the  throughput  at  queue  /.  A  .  in  units  of  A,-,  the  throughput  of  queue  /*.  viz 

A  =  tf,A  •.  /  =  1.  2.  .  V 

(3.18) 

If  i  measures  the  system  time  across  queue  /.  then  obviously  the  average  time  a  customer 
needs  between  two  successive  departures  from  queue  /*  is  given  by  the  sum  of  the  products  of 
the  form  [average  number  of  visits!  x  [mean  wailing  lime),  namely 

2>' 

-i 

(3.19) 

Tor  simplicity  of  notation,  define  A  "  =  A  •  Now  equations  (3.13)  to 

(3.15)  translate  into 

/  ( X  )  =  t  [1  +  //  ( X  —  1 )  1 

(3.20) 

A  MX)  =  X/£  #  /  <X> 

(3.21) 

//  (X)  =  A*<X  i  (A  ) 

(3.22) 

Define  /  =  H  /  and  as  usual  traffic  intensities  p  =  H  r  .  Then  we  find 

i 

+ 

a 

it 

(3.23) 

A  ‘  =  X/£/<X  > 

(3.24) 

and 

n (X  )  =  A  *  <  X  )  /  ( X  ) 

(3.25) 

Equations  (3.12)  anti  *3  23)  to  (3.251  constitute  the  recursion  to  evaluate  all  quantities  of 
S <  At  )  with  general  routing  They  are  of  the  same  form  as  (3.13)  to  (3.15). 
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Next  we  shall  discuss  the  multi-chain  case.  We  assume  that  there  are  R  closed  routing 
chains.  We  will  use  superscripts  r  (/ •  =  I,  2.  ,R  )  to  denote  that  a  given  quantity  belongs 

to  chain  r.  We  do  not  consider  the  case  where  customers  change  class  membership  within 
chains.  Note,  however,  that  as  shown  in  Eq.  (3.13),  this  is  only  a  trivial  reduction  of  general¬ 
ity.  Also  we  shall  drop  the  asterisks  used  in  (3.23)  to  (3.25).  As  before,  we  assume  constant 
service  rales.  We  introduce  the  following  notation: 

A  =  (A '.A'1.  ■  ,AW)  .  population  vector, 

A-c,  =  (A1, A2,  .A'  \A'-I.  ,kK):  population  vector  with  one  less  customer 

in  chain  r, 

R(i):  Set  of  chains  visiting  queue  /, 

Q<r):  Set  of  queues  in  chain  r. 

A' :  Throughput  of  marked  queue  in  chain  / , 

r  Mean  service  time  of  a  chain  /  customer  in  queue  /, 

i1'.  Mean  time  a  chain  /  customer  spends  at  queue  i  between  successive  visits  to  the 
marked  queue  of  chain  /  . 

ii':  Mean  number  of  chain  /  customers  at  queue  /. 

n  =  I,  ii':  Mean  queue  si/e  of  queue  /. 

f> '  =  W,V:  Traffic  intensity  of  chain  /  at  queue  i , 

Mean  number  of  visits  a  chain  /  customer  makes  to  queue  /  between  successive 
visits  to  an  arbitrarily  marked  queue  in  chain  r. 


The  mean  waiting  time  equations  follow  from  Theorem  I. 

/'<  A  )  =  +  »'  1,7  //.'( A— e, )  (3.26) 

The  first  term  in  Eq.  (3.26)  is  the  customer's  own  service  demand  and  the  sum  measures  the 
backlog  of  work  he  encounters  upon  arrival. 

The  recursion  now  follows,  analogous  to  (3  13)  to  (3.15)  or  (3.23)  lo  (3.25)  as. 


1 

+ 

II 

< 

(3.27) 

A '  <  A  )  =  A  7  £  r 

(3.28) 

6  1  *  1 

a  ( A  )  =  ^  A'  ( A  )/'<  A  ) 

(3.29) 

■  t  K  I  i 


where  (as  usual)  /  =  1 .  2.  .  V:  /  =  1 .  2.  .  R  ,  A  >0  ;  and  </'(0>  -  0 
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Equations  (3.27)  and  (3.29)  allow  an  easy  recursive  evaluation  of  unknown  quantities. 


3.4.3  The  Heuristic  Solution 


The  complexity  of  recursion  (3.27)  to  (3.29)  for  both  storage  and  operation  is  of  the 

order 


n 


r-l 


(3.30) 


Obviously,  it  will  be  impossible  to  solve  for  more  than  a  small  number  of  chains  (clearly  less 
than  10).  The  following  heuristic  method,  derived  from  the  mean  value  analysis,  overcomes 
the  complexity  barrier  of  Eq.  (3.30). 


We  note  that  the  recursion  is  the  source  of  the  complexity.  It  is  the  goal  of  the  heuris¬ 
tic  method  to  replace  the  recursion  by  an  iteration  that  is  performed  at  the  point  K  only.  We 
define  quantities  £'  as  follows 

f,'(K)  =  n,(K)  -  n,(K-et)  (3.31) 

Thus,  £'  measures  how  the  customer  added  to  S(K-e,)  is  distributed  over  the  individual 
queues. 


Obviously  we  have  (all  i  =  1,  2,  ...  .  N.  r  =  1,  2 . R) 

i 

.-i 

and  also 


<K£,'<1 


(3.32) 

(3.33) 


Suppose  now  that  we  had  a  function  that  would  yield  £'  in  terms  of  quantities  of  S(K), 
for  example 

£.'  =  Hi.  r.  Ia'.  /  =  I.  2.  •  •  •./?))  (3.34) 

where  we  have  omitted  the  arguments  denoting  the  chain  population.  We  may  now  rewrite 
Eqs.  (3.27)  to  (3.29).  again  dropping  the  arguments  (A).  as  follows 


e;  =  / (/.  Ia'I) 

(3.35) 

r;=  p,1l+/f,-f,1 

(3.36) 

a'  =  Kr  /  L  /; 

(3.37) 

<€  Qir) 

",  =  X  A'// 

(3.38) 

We  have  now  obtained  a  nonlinear  system  of  equations  that  is  independent  of  K,  the 
source  of  the  high  operation  count  in  Eq.  < 3 .30) .  We  would  solve  Eqs.  (3.35)  to  (3.38)  by  a 

simple  iteration  method,  starting  with  initial  values  for  n,  and  K'  (i  =  1,  2 . N.  r  =  1,  2, 

...  ,  R)  and  then  iterating  through  Eqs.  (3.35)  to  (3.38)  in  a  cyclic  fashion  until  convergence  is 
obtained  (or  divergence  established). 
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: 


In  its  exact  form,  the  function  (3.34)  is  no  less  complex  than  the  original  problem. 
Hence,  it  is  our  goal  to  obtain  an  efficient  heuristic  for  Eq.  (3.34).  The  proposed  heuristic  will 
coalesce  the  R  chains  into  a  single  chain  that  can  be  solved  efficiently  by  the  recursion  (3.23)  to 
(3.25). 


If  a  customer  is  removed  from  a  ~hain,  then  all  of  the  values  n,r, 
i  =  1,  2,  ■  •  ■  ,  ;V,  -  =  1,  2,  ,  R  are  affected.  However,  since 

X  «/(*')  -  nj(K—er)  =  0  for  all  /  ^  r  (3.39) 

•  t  <?(,) 

(where  r  is  the  chain  with  one  less  customer)  we  may  assume  that  £,'  is  affected  mostly  by 
chain  /•.  Therefore,  we  estimate  from  a  single  chain  problem  with  redefined  parameters. 
The  capacity  of  queue  i's  server  devoted  to  chains  j  —  1 .  2,  •••,/?,/  ^  r  is  given  by 

X  A'r/ 

where  the  sum  is  over  7  =  1,2,  •  ,  R  but  i  ^  r  .  Taking  the  point  of  view  of  the  fluid 

approximation  of  queueing  systems,  we  may  agree  that  a  chain  r  customer  "sees"  a  server  with 
a  reduced  rate  given  by  Eq.  (3.40)  below.  Thus  his  mean  service  limes  are  not  t'  but  adjusted 
values 


( 1  +k' ~r  -  X  A'r/) 

itm<) 


(3.40) 


Suppose  that  h'(K)  (i  =  I,  2,  -  •  •  ,  ,V)  are  the  mean  queue  sizes  of  a  single  chain 
queueing  problem  with  population  K  and  with  parameters  given  by  Eq.  (3.40).  Then  we  set 

£/=  h.UC)  -  «,<A'-1)  (3.41) 

h,(Kr)  and  )  are  obtained  from  Equations  (3.12)  to  (3.15).  This  completes  the  heuris¬ 

tic  method. 


now 


The  computational  complexity  of  one  iteration  step  through  Eqs.  (3.35)  to  (3.38)  is 


R 

I 


Kr 


(3.42) 


clearly  an  affordable  effort  even  for  a  large  number  of  closed  chains  with  sizable  populations. 


Note  that  the  heuristic  method  provided  by  Reiser  has  reduced  the  computational  com¬ 
plexity  tremendously.  For  an  R -chain  network,  if  N,  is  the  number  of  queues  in  the  /'*  chain, 
then  the  original  complexity  in  determining  G  is 

|/V,  -t-  K,  -  1 

\  N,  -  1 

where  K ,  is  the  total  number  of  tokens  in  chain  i.  The  above  parenthesis  represents  the  total 
number  of  ways  to  put  Kt  tokens  in  N,  queues.  With  Reiser's  heuristic  method  this  complexity 
reduces  to  that  in  Eq.  (3.42). 


R 

n 


We  leave  Chapter  3  here  without  giving  an  example  of  applying  this  control  scheme. 
Dr.  Reiser  gives  examples  in  great  detail  and  therefore  we  need  not  repeat  the  same  effort  here. 
The  interested  reader  may  refer  to  {REIS  78). 
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CHAPTER  4 

STATIC  FLOW  CONTROL  BASED  ON  LIMITING 
THE  PERMIT  GENERATION  RATES 

In  this  chapter  we  present  a  new  class  of  How  control  techniques.  Two  schemes  are 
proposed  and  their  models  analyzed.  Both  are  based  on  limiting  the  permit  generation  rates  of 
logical  channels.  It  is  found  that  the  second  model  gives  a  performance  very  close  to  the  ideal. 
Both  schemes  are  very  easy  to  implement,  more  importantly  they  involve  no  computational 
complexity  (one  can  calculate  the  entire  network  performance  on  the  back  of  an  envelope)  and 
other  network  features  such  as  "piggy-back”  acknowledgements  and  "time-outs"  can  easily  be 
incorporated  in  the  models.  It  turns  out  that  the  schemes  can  be  modeled  by  Jackson's  open 
network  of  queues  (JACK  631  and  the  network  of  queues  presented  by  Basket!  et  al.  IBASK 
75].  The  first  scheme  serves  as  an  introductory  one  which  has  an  inherent  shortcoming.  The 
second  scheme  overcomes  this  shortcoming  and  appears  to  be  an  "ideal"  flow  control  method. 

4.1  The  Currency  Circulation  Model  -  Scheme  1 

The  name  of  this  scheme  comes  from  the  analogy  between  the  way  that  the  scheme 
controls  the  How  in  a  network  (by  properly  generating,  circulating  and  destroying  permits)  and 
the  way  that  our  society  controls  the  flow  of  commodities  (by  issuing,  circulating  and  destroying 
currency).  Fortunately  we  manage  to  resist  the  temptation  to  inflate  the  currency! 

4.1.1  Description  of  the  Scheme 


COMPUTER  COMMUNICATION 
NETWORK 


Consider  the  computer  communication  network  shown  in  Figure  4.1  This  network 
may  be  described  as  follows: 
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(1)  The  network  consists  of  a  number  of  nodes  inter-connected  by  communication  chan¬ 
nels. 

(2)  There  is  an  external  source  of  permits  which  generates  permits  at  a  rate  g.  Assume 
that  the  generation  process  is  Poisson.  Further,  assume  these  permits  are  distributed  to 
permit  queue  i  with  a  probability  r,.  Permit  queues  are  associated  with  physical  chan¬ 
nels.  For  example,  permit  queue  i  is  associated  with  the  physical  channel  /  which  con¬ 
nects  node  -v  to  node  y.  The  permits  in  queue  /  are  to  be  used  by  messages  which  ori¬ 
ginate  at  node  v  and  are  routed  through  node  y  as  the  first  hop  on  the  way  to  their 
eventual  destination. 

(3)  An  externally  arriving  message  at  a  node  must  secure  a  permit  from  the  corresponding 
permit  queue  at  that  node  before  it  can  be  launched  into  the  network  on  the  first  leg  of 
its  journey.  If  the  message  finds  no  permit,  it  is  immediately  rejected  and  "lost".  Note 
that  externally  arriving  messages  may  not  queue  at  their  origin  while  awaiting  entry  into 
the  net.  Once  a  message  is  accepted  into  the  net  it  is  stored-and-forwarded  over  some 
fixed  route  to  its  destination.  We  assume  that  messages  arrive  at  logical  channel  jk 
(origin  node  destination  node  k)  according  to  a  Poisson  process  at  rate  ylk.  (A  logi¬ 
cal  channel  is  associated  with  each  origin-destination  pair). 

(4)  There  is  also  a  sink  which  destroys  any  permits  which  find  their  way  to  the  sink. 

(5)  Alter  launching  a  message  into  the  network,  a  permit  at  queue  /  is  then  immediately 
passed  to  a  neighbor  permit  queue  ./  with  probability  r„  (where  queue  ./  could  be  the 
sink). 

(6)  To  prevent  stalling  of  extra  permits  at  queue  /  when  the  message  traffic  is  low,  we  have 
a  basic  circulation  rate  n,Q  to  pump  the  permits  out  of  queue  /.  That  is.  when  there  is 
zero  message  traffic,  the  permit  at  the  head  of  the  permit  queue  /  will  remain  there  for 
an  exponentially  distributed  length  of  time  with  mean  Un,0  (after  which  it  will  move  to 
neighbor  permit  queue  /  with  probability  r,,).  This  basic  rate  is  added  to  the  permit  cir¬ 
culation  rate  because  of  message  arrivals. 

Before  going  any  further,  let  us  give  an  example  to  see  how  the  control  scheme  can  be 
modeled  for  a  network. 

Figure  4.2  shows  a  simple  5-node  computer  communication  network.  Compare  this 
with  the  permit  queue  model  in  Figure  4.3.  Note  that  we  have  ten  physical  communication 
channels  ( i.e. .  five  full-duplex  channels)  and  twenty  logical  (origin-destination)  channels.  Each 
physical  channel  corresponds  to  a  permit  queue.  In  Figure  4.3  we  have  a  conceptual  source  and 
a  conceptual  sink.  The  permits  generated  at  the  source  (with  rate  g)  are  distributed  to  queue  / 
with  probability  r,.  Permits  at  queue  /  are  removed  at  a  rate 

Vr  =  M,„  +  L  V' 

rC  O 

where  y,  is  the  external  message  arrival  rate  of  logical  channel  r  and  O,  is  the  set  of  logical 
channels  which  originate  at  v  and  are  routed  through  y  as  their  first  hop  enroute  to  their  desti¬ 
nation  (Nodes  x,y  are  connected  by  physical  channel  i.  See  assumption  (2)  in  this  section). 

After  its  removal  from  queue  i.  a  permit  enters  queue  /  with  probability  r„. 
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Figure  4.2.  A  simple  computer  communication  network. 


Figure  4.3.  The  currency  circulation  model  of  the  simple  network  in  Fig.  4.2. 


As  an  example,  lei  us  assign  the  following  values  to  the  r,'s  and  r„' s: 
r,  =  0.1  V queue  /,  /  =  1.2.  .10 


where  //,  =  !+  the  number  of  adjacent  queues.  (The  extra  "one"  is  for  the  sink.)  We  shall 
continue  this  example  later. 

4.1.2  The  Queueing  Model  for  Permits 

Let  y,*  be  the  external  input  rate  of  messages  to  logical  channel  ik  (i.e.,  the  logical 
path  between  node  /  and  node  k).  The  external  arrival  rate  of  permits  at  queue  /  is  g,  =  gr, 
and  the  departure  rate  of  permits  at  queue  /  is 

m,.  =  +  y,  <41) 

where  y,  4  £  y,. 

rtO, 

The  total  arrival  rate  of  permits  to  queue  /  (from  outside  the  network  and  from  other 
queues)  is 

kn  =  gr,  +  £  A (4.2) 

,-i 

where  g  is  the  total  permit  generation  rate. 

Thus,  we  have  a  Jackson-type  network  of  M  queues  for  permits.  The  joint  probability 
for  the  number  of  permits  distributed  at  all  of  the  queues  is  then  (KLE1  75] 

Pn\»\.ii2 . "tr!  =  Pr[{"\)Pr2{"2)  ’  pny,{"»)  <4  3) 

where  //,  denotes  the  number  of  permits  in  queue  /.  and  Pp  ( n, )  is  the  probability  that  queue  / 
has  //,  permits.  Pp  < n, )  is  obtained  from  the  solution  to  an  M/M/1  queue  with  input  rate  Kr  as 
in  Eq.  (4.2)  and  output  rate  nP  as  in  Eq.  (4. 1 ).  i.e.. 

Pn  (n,)  =  <  1  —  p ,,  )pP  where  p,,  =  kr/nP 

Observe  that  the  permits  do  not  necessarily  enter  or  leave  a  queue  according  to  a  Pois¬ 
son  process.  However,  we  shall  assume  this  (and  point  it  out  as  we  do  so)  whenever  it  is  con¬ 
venient 

4.1.3  The  Queueing  Model  for  Messages 

Observe  that  each  logical  channel  r  is  associated  with  a  queue  at  its  originating  node 
(e  g.,  in  Figure  4.3,  logical  channel  5-4  is  associated  with  queue  10  at  node  5). 

Denote  the  throughput  (message/sec)  of  logical  channel  r  as  Sr.  Then. 

V,  =  U-P,,(0)!y,  -  —  y,  -  — ^ -  y,  (4.4) 

Mr  M.0  +  y 

where  Pp  (0)  is  the  probability  of  no  permit  in  queue  /  such  that  r€  O 

After  entering  the  network,  a  message  is  routed  through  a  set  of  nodes  and  physical 
channels  to  its  destination  Upon  arriving  at  a  node,  the  message  joins  the  message  queue  asso¬ 
ciated  with  the  next  physical  channel  in  its  path,  and  waits  there  for  transmission  through  the 
channel. 


We  describe  the  state  of  the  system  for  messages  by  using  "customer  classes"  as  in 
[BASK  75],  Each  class  corresponds  to  a  logical  channel.  In  particular,  consider  the  vector  of 
column  vectors 


where 

(a)  M  is  the  total  number  of  physical  channels 

(b)  R  is  the  total  number  of  logical  channels 

(c)  Column  vector  /  denotes  the  state  of  message  queue  /  corresponding  to  physi¬ 
cal  channel  / 

(d)  K,r  (r  =  1,2,  •••,/?)  denotes  the  number  of  messages  at  queue  /  from  logi¬ 
cal  channel  r ,  regardless  of  their  actual  relative  positions  with  respect  to  other 
messages  in  queue  i. 


If  we  assume  that  the  messages  enter  the  network  from  a  Poisson  process,*  then 


■  IK 


A  V- 


(iu  ,,-Vi-n 

,„|  I-  *<R 


mC 


where 


G  is  the  appropriate  normalization  constant,  and 

R 

A,  =  I  A, 

r  —  I 

C,  is  the  service  rate  of  channel  i  (bits/ sec). 


(4.5) 


(4.6) 


\/ n  is  the  average  message  length  (bits) 

which  is  assumed  to  have  an  exponential  distribution. 


and 


,  S,  ,  if  messages  of  logical  channel  r  are  routed  through  physical  channel  / 
~  \  0,  otherwise 

We  also  define  0n  =  I,  for  convenience. 


*  It  is  easy  to  prove  that  in  our  system,  if  permits  arrive  at  permit  queues  from  Poisson 
processes,  then  messages  enter  the  network  with  Poisson  processes  Unfortunately,  as 
mentioned  in  Sec.  4.1.2.  permits  do  not  arrive  at  queues  from  Poisson  processes  and  hence 
messages  do  not  enter  the  network  with  Poisson  processes. 
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Let  p„ 


ii 


mC 


for  (  =  1,2,  ...  Af,  r  =  1 ,  2 . /?.  Then, 


Kir 


^4/t 


1  MR 


M  A-  '  R  * 

=  G  n  *  r tv  ,  n  p* 

,-|  *,!•  *,R  r-1 


Therefore, 


M 


*,! 


G  I  n  k ,,!  •  K,*!  n  p» 


all  feasible  stales  /*! 
M 


M  00  f, 

=n  i  p,i + p.2  +  •  -  +  p,r 

/-I  K- 0 


The  last  equality  can  be  justified  by  proving  that  for  any  term  in  the  right  hand  side,  there  is  an 
equivalent  term  in  the  left  hand  side  and  vice  versa.  For  example,  [J  t>\  is  a  term  belonging 

-i 

to  the  right  hand  side,  but  it  corresponds  to  the  feasible  state 


0)  (0 

and  therefore  it  is  a  term  in  the  left  hand  side. 
Define 


P.  =  L  P' 


r-l 


Then 


G-'-n  =n~r 

.-I  K  -0  .-I  1  P< 


i.e.. 


M 


G  =  n  u-p,) 


(4.7) 


Let  P,(K,)  denote  the  marginal  probability  that  the  /'*  physical  channel  has  K,  messages,  then 
[BASK  75] 


OO  OO  <30 


W>=I  I  I  I  Gp*  p*'  =  0  -p,)p/ 

A,’, -l  A  | ™ I  A  . i ™ l  A 


(4.8) 


Thus  the  system  is  completely  solved  Moreover,  from  J.  W.  Wong’s  work  for  the 
Laplace  transform  of  the  density  for  the  delay  of  class  r  messages  with  fixed  routing  (under 
some  restrictions)  [WONG  78 j 


T'i  i  =  n  (>  ~p,)  . 

r  +pCM  -p  > 


(4  9)* 
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where  /(/  >  =  {queue  /  I  queue  ;  is  in  the  path  of  logical  channel  /). 


Thus,  the  average  delay  is 


and  the  variance  of  delay  is 


/;=  I 

t/i,  i 


i _ 

M(  u  -p,) 


"l-  I 

( n, ) 


_ ] _ 

lMCU-p,)l2 


(4,10) 


(4,11) 


4.1.4  Result  of  a  Simple  Example 

We  apply  the  above  scheme  to  the  network  shown  in  Figures  4.2  and  4,3.  The  calcu¬ 
lated  throughput  and  the  simulated  throughput  are  both  plotted  in  Figure  4.4,  where  we  have 
(see  Sec.  4.1.1 ) 

X  =  30  (total  generation  rate  of  permits) 
r,  =  1/10 

r„  =  —  =  [1  +  number  of  queues  adjacent  to  queue  /) 

n, 

and  n,Q  =  5,  V  /  (the  basic  circulation  rate). 

With  these  parameter  values,  we  have  (see  Eq.  (4.2)) 

10 

£  Kr  =  96 
-1 

We  observe  in  Figure  4.4  that  the  asymptotic  value  of  total  throughput  is  also  96.  This  is  not 
an  accident  and  we  will  explain  why  in  the  next  section.  Also  note  that  we  did  not  obtain 
results  in  the  low  input  region  since  in  that  region  permits  enter  a  queue  faster  than  they  are 
removed  (with  the  basic  circulation  rate  m  „  ~  5  fixed),  i.e..  we  obtained  an  unstable  system  for 
the  permits.  In  fact,  if  p,(|  ^  Kr  for  any  i.  this  will  occur.  This  is  one  of  the  problems  which 
motivated  us  to  investigate  the  second  scheme  in  Section  4.2. 

4.1.5  Some  Observations  of  the  Model 

(I)  Upper  limn  ol  ihv  thnmxhpw:  Let  y ,  denote  the  external  message  input  rate  of  logical  chan¬ 
nel  r .  Let  y ,  =  £  y,  where  O,  is  the  set  of  all  logical  channels  whose  messages  are  routed 

r€  o 

through  physical  channel  i  as  the  first  hop.  Then  the  total  throughput  of  those  logical  channels 
is 

S,-H-Ppm\y,  (4.12) 


This  equation  holds  only  for  looping  free  and  non-merging  after  splitting  routing  schemes. 


TOTAL  THROUGHPUT 
(messages/sec) 


(4.13) 


Here  we  have  made  use  of  Eq.  (4.4),  and  kPj  is  a  constant  determined  by  Eq.  (4.2).  Note  that 
as  y j  —  «>,  Sj  — •  .  This  shows  that  our  scheme  does  put  a  ceiling  (£  kp)  on  the  accep- 

i 

tance  rate  of  the  external  input  to  protect  the  network  from  becoming  overloaded.  This 

explains  why  the  asymptotic  value  of  the  total  throughput  shown  in  Figure  4.4  is  96  which  is 

10 

L  */>,• 

/-i 


(2)  Inefficiency  of  the  scheme:  In  order  to  prevent  permits  infinitely  accumulating  in  a  queue 
when  external  traffic  is  near  zero,  we  must  let 


Vj 


Since 


we  have 


Sj  -*  y,  as  yj  —  o 

which  approaches  zero  linearly  with  yr  And  when  -y,  -  kp.jQ ,  A:  —  1,  2,  •  • 


kfij 


'  -  kn,0  +  *  +  l  ^ 

Figure  4.5  shows  this  family  of  performance  curves  for  this  control  scheme. 


We  see  that  the  best  we  can  do  (without  the  danger  of  having  an  infinite  number  of 
permits  at  a  queue  so  that  the  arriving  messages  thereafter  are  not  flow  controlled)  is  to  let 
nJg  >  kp  Therefore  the  best  performance  we  can  obtain  is  the  uppermost  curve  under  the 
ideal  one  (wi,l>  hroughput  =  \pf  2  when  input  —  ^  ,  V  j).  This  is  apparently  not  satisfactory 

because  it  is  too  conservative.  In  the  next  section  we  will  attack  this  weak  point  and  obtain  a 
nearly  perfect  static  control  scheme. 


(3)  The  bottleneck  of  a  network:  If  we  define  the  bottleneck  of  a  network  as  the  physical  channel 
whose  associated  queue  has  the  largest  average  backlog,  then  we  are  able  to  identify  the 
bottleneck  with  respect  to  a  given  set  of  network  parameters.  Let  N,  denote  the  backlog  at 
message  queue  / .  Then 

max  £[/V,]«  max  J  K,P,(K,)  (see  Eq.  (4.8)  for  P,(K,) 

* -i 


-  max  (l  —  p , )  £  K,pK 

K,-\ 


Figure  4.5.  Performance  of  the  currency  circulation  model. 


I 

=  max  - - - 

I  -P 


but. 


t>=  z 

/ 1  K  I  I 


■V, 


where 


R(i)  =  (logical  channels  using  physical  channel  / )  .  Therefore. 


max  /:  I V  1  =  max 


I 

1-  L  S,/fiC\ 


where  .V,  is  "iven  in  Eq.  (4.4). 


4.2  A  More  Interesting  Model  -  Scheme  2 


The  control  scheme  described  in  Section  4.1  has  some  weak  points.  First,  the  perfor¬ 
mance  curve  is  far  below  the  ideal  one  (see  Figure  4.5).  Second,  we  do  not  know  how  to 
optimally  specify  the  transition  matrix  for  permits.  In  this  section  we  look  into  a  more  interest¬ 
ing  scheme  which  produces  much  more  desirable  results. 

4.2.1  The  Model  Description 


This  scheme  works  similar  to  the  one  in  the  last  section  except  that: 


( 1 )  After  launching  a  message  into  the  network,  a  permit  is  tlcurowtl  immediately  instead  of 
being  circulated  to  other  permit  queues 


t 


(2)  Permits  for  logical  channel  r  are  generated  (from  a  Poisson  process)  at  rate  g,  and  are 
queued  in  permit  queue  qr.  Note  that  there  is  a  permit  queue  for  each  logical  channel. 

(3)  We  have  a  finite  buffer  size  Nr  for  each  permit  queue  qr  When  the  buffer  is  full,  the 
newly  generated  permits  are  destroyed  thus  controlling  the  unlimited  growth  of  permit 
queues.  The  permit  queue  is  shown  in  Figure  4.6a. 


maximum  Queue  size 


Figure  4.6(a)  Permit  queue  r  for  logical  channel  r. 


4.2.2  Analysis  of  the  Scheme 

The  model  for  permits:  Each  permit  queue  q,  (for  logical  channel  r)  can  be  described  by 
the  finite  Markov  chain  in  Figure  4.6b. 


Figure  4.6(b)  The  Markov  chain  describing  the  permit  queue. 

Let  Pkr  —  P[k  permits  in  qr].  We  then  have 

Pk  -  [—1  P0  for  k  -  0.  1,  2,  •  ,  N,  (4.14) 

r  l?' I 

where  p,  £  £,/y,,  and 
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N 

y, 


(4.15) 


Pn  = 


1  -  Ofr/y,) 

1  -  (g'/y,)”’*1 


(y  r  Kf  ) 


(y,V,+l  -  /'+') 


Note  that  for  nr  >  0,  as  yr  — *  0  (i.e.,  the  message  input  rate  is  near  zero) 

p0,  0 

That  is,  qr  is  always  full. 


As  y,  ~ *  k,  we  have 

P  —  — - - 

N,  +  1 

As  y,  — -  A,g,  for  A  =  2,  3,  •  •  •  we  have 

„  1  +  1 A 

*<»  V  +  1 

I +  <!/*)' 

and,  as  A  —  oo  (j.e.,  the  message  input  rate  is  very  large)  P0  — -  1. 


(4.16) 


(4.17) 


(4.18) 


Figure  4.7  The  non-empty  probability  of  a  permit  queue. 


In  Figure  4.7,  we  plot  I  -  P„  (the  non-empty  probability  of  permits  for  logical  channel 
r)  versus  yr.  Observe  that  1  -  Pn  at  y,  =  .?,  is  larger  than  0.9  at  .V,  =  10.  This  means  that  less 
than  one  out  of  ten  messages  is  rejected  for  lack  of  permit.  The  total  throughput  for  logical 
channel  r  is 


■V,  =  1 1  -  P„ 


Pr~Pr 

yr  =  - - TT7  y, 

—  n 


(4.19) 


For  large  .V,,  as  yr—<x>  <i.e.,  heavy  applied  load  for  logical  channel  r )  we  have 
p,  —  0,  and 


..  _  & 

V  =  —  y,  =  a, 

y, 

as  yr  —  we  have  p,  —  1,  and 

-v.  - > 

At,  + 1 


(4.20) 


(4.21) 


and  as  y,  —  0  (i.e.,  almost  zero  applied  load  for  logical  channel  r )  we  have  p,  —  and 

V,  -  y,  (4.22) 


4.2.3  Result  of  the  Simple  Example 

In  Figure  4.8,  we  show  the  result  of  applying  the  scheme  to  our  example  network  (Fig¬ 
ure  4.2)  with  ,c,  =*  5.  The  plot  is  for  total  throughput  versus  input  rate  per  logical  channel. 

We  see  that  we  can  bring  our  performance  curve  as  close  to  the  ideal  curve  as  we  desire 
by  letting  S,  be  sufficiently  large.  This  is  an  extremely  desirable  property.  Also,  the  "danger" 
mentioned  in  Sec.  4.1.5  is  avoided  here.  That  is.  an  idle  logical  channel  can  no  longer  make 
itself  uncontrolled  later  (because  of  the  large  number  of  permits  accumulated  during  the  idle 
period).  The  computer  communication  network  can  now  be  viewed  as  a  network  of  queues 
with  external  input  .S',  (which  is  assumed  to  be  a  Poisson  process)  and  with  some  routing 
scheme  (fixed  for  our  case).  The  delay  can  easily  be  obtained  through  the  application  of  J. 
Wong's  delay  equations.  The  network  model  is  thus  completely  solved.  Observe  again  that  S, 
is  not  a  Poisson  process,  but  it  becomes  Poisson  asymptotically  when  Nr  — •  <».  We  assume  S, 
is  Poisson  in  the  following  two  sections. 

4.3  The  Piggv-back  Acknowledgement  Scheme 

The  "ideal"  How  control  scheme  in  the  previous  section  operates  under  the  simplified 
assumption  that  the  transmission  channels  are  noiseless  and  thus  neither  time-out  nor  ack¬ 
nowledgement  schemes  are  necessary.  We  now  relax  this  assumption  to  obtain  a  more  realistic 
model.  As  the  assumption  is  removed,  a  destination  will  issue  an  ACK  for  each  message  it 
receives  and  a  source  will  retransmit  a  message  when  the  ACK  for  it  does  not  return  in  the 
allotted  time.  The  ACKs  are  usually  piggy-backed  by  messages. 

We  first  look  into  the  piggy-back  acknowledgment  schemes.  In  order  to  maximize  the 
throughput,  a  computer  communication  network  usually  arranges  the  acknowledgements  for 
messages  in  one  flow  to  be  piggy-backed  by  the  messages  in  the  opposite  flow.  Yet  the  traffic  in 
the  two  mutually  opposite  flows  is  usually  not  symmetric,  so  that  we  usually  allow  a  message  in 
one  flow  to  simultaneously  piggy-back  ACKs  for  many  messages  in  the  opposite  flow.  To 
prevent  an  indefinite  delay  of  ACKs  (which  may  happen  when  the  opposite  flow  is  idle),  we 
usually  set  upper  limits  both  on  the  number  of  ACKs  accumulated  (waiting  to  be  piggybacked) 
and  on  the  time  that  the  first  ACK  has  spent  waiting  for  a  message  to  pick  it  up.  If  these  limits 
are  reached  before  a  message  comes  up,  a  special  control  packet  is  generated  to  pick  all  the 
waiting  ACKs.  We  first  investigate  the  case  where  we  set  a  limit  only  on  the  number  of  ACKs 
waiting  for  piggyback.  Then  we  look  into  a  more  complicated  case  where  limits  on  both  the 
number  of  waiting  ACKs  and  the  time  that  ACKs  have  been  waiting  are  imposed. 
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performance  curves  of  Scheme  2. 


4.3.1  Piggy-back  Scheme  with  Limit  on  Number  of  ACKs  Waiting  to  be  Piggy-back.d 

Consider  a  computer  communication  network  as  shown  in  Fig.  4.9.  The  logical  chan¬ 
nels  S-D  (the  forward  channel)  and  D-S  (the  backward  channel)  have  external  message 
input  rates  y, ,  and  yh.  Only  a  portion  of  them.  S,  and  Sb ,  can  be  accepted  by  the  network 
(because  of  the  limited  permit  generation  rate). 


Figure  4.9  Model  of  piggy-back  scheme  with  limit  on  number  of  waiting  ACKs. 


The  accepted  messages,  say  the  S-D  messages,  are  routed  through  the  network  to  the 
destination  D.  The  ACKs  for  these  messages  then  enter  a  queue,  Qb.  where  they  are  picked  up 
by  the  D-S  messages.  If  L  ACKs  are  accumulated  in  Qb  before  a  D-S  message  comes  up, 
then  node  D  generates  a  special  control  packet  to  carry  these  L  ACKs.  These  control  mes¬ 
sages.  together  with  Sb ,  make  the  effective  backward  traffic  S^.  Among  S^,  Sb  need  to  be 
acknowledged  by  node  S.  By  assuming  S, ,  and  Sb  are  Poisson  processes  Qb  can  be  described 
by  the  following  Markov  chain.  (See  Figure  4.10.)  Let  Pk  A  P[A  ACKS  in  Then 


Pk 


Sf 

5,  +  S, 


/t-i 


1  A  ^  Z.-1 


(4.23) 


Let 


then 


p»  k 


s. 


Sr  +  Sb 


(4.24) 


Pk  =  pH  Po 


(4.25) 


Thus, 


Po  = 


_ _ I _ 

•  +  f>b  +  ■  ■  ■  +  Pb~l 


1  ~Pb 
1  ~Ph 


(4.26) 
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Figure  4.10  The  slate  diagram  for  Qb  in  Figure  4.9. 
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(4.27) 


Note  that  k  can  not  be  equal  to  L  because  a  special  control  packet  is  generated 
immediately  once  there  are  L  ACKs.  Let  us  now  calculate  Tbs  the  average  time  an  ACK  waits 
in  Qb.  As  shown  in  Fig.  4.11,  a  random  tagged  customer  (an  ACK  for  an  Sr  message)  arrives 
at  Qb  at  time  r  and  sees  that  K  ACKs  are  already  there.  He  must  wait  at  Qb  until  either  the 
L'h  ACK  arrives  or  until  a  D-S  message  is  accepted  by  the  network  (which  picks  up  all  the 
ACKs). 


ACKs 

PICKED  UP  BY  A  D-S  MESSAGE 
OR  SPECIALLY  GENERATED  MESSAGE 


tagged 

CUSTOMER 


TIME 


Figure  4.11.  The  time  period  for  a  tagged  ACK  to  wait  in  Qb. 

Let  P[Tb  -  /|K]  ^  Prltagged  customer  waits  in  Qb  for  t  sec  >  0  given  he  found  K 
ACKs  in  Qb  for  0  ^  K^L— 21.  Then  we  have. 
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f 


P[Th  =  t\K\dt  = 


(S,i) 


L-K-2 


(L  -  K  -  2)\ 


_v,  _i-,  L~«-2  (S,t)' e  Sf' 

S,  di  e  V  +  .S',  di  v  *  X  — ~i - 

,-o  J  ■ 


a(u/) 


(Z.  — AT-2) ! 


Pa  *  1 


l-K-2 

+  I  <1 

/-0 


Pa>p/ 


a(ai)1  e~ 


dr 


(4.28) 


where  a  ^  S',  + 


The  first  term  of  the  above  equation  means  that  the  Eth  ACK  arrives  at  (/,  t+dt)  and 
before  any  D-S  message  arrives;  the  second  terms  means  that  a  D—S  message  picks  up  all  the 
ACKs  at  d,  t+dt)  and  before  the  Eth  ACK  arrives. 


The  conditional  expected  value  of  Th  is 

oc 

ElTh\K\  =  J  ,P[Th  =  r\K  ]  dr 

0 


=  p/-x-i  +{l_(Jh)L-£2p/tl±l 

«  t-o  « 


=  J-  [l+2pA+3p2  •  +  IL-K-Dpt;-*-1] 


-±[ph  +  2Pl+  •••  + 


—  li  +  Pb  +  pI  +  ■  •  •  +  pb  K~2\ 

a 


1  1-Pa 


L  -A' -I 


a  1  -p. 


(4.29) 


We  have  used  the  mathematics  developed  in  Appendix  A  in  deriving  Eq.  (4.29).  The  uncondi¬ 
tional  expected  value  of  Th  is 


L-2  L-2  i  1  -of-'1'1  ( 1  —  Oh)  Ok 

E\Th\  ~  ±  E[Th\K)PK  ~  ±  ,P_* 

A-0  A-0  a  *  PA 


1  — Pa 


1 


fe2  Pa'- Pa -1 


a  '  k—o  1  — Pa 

-rfi-^{[l+PA+  +pAt-2l-(T-l)pAt-') 

1 


ad -Pa") 


(4.30) 
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We  have  used  Eq.  (4.27)  and  the  fact  that  /:  \Th\  E-ll  =  0  in  deriving  Eq.  (4.30) 


Note  that  as  Sh  —  0,  i.e.. 


From  Eq.  (4.30),  we  have 


the  D-S  traffic  — >  0, 


V, 

.S',  +  Sh 


1 


t'frj 


J_  L  ~ 1 

S',  2 


(4.31) 


This  is  intuitively  true  because  when  Sh  —  0,  the  only  group  output  from  Qb  is  caused 
by  the  arrival  of  the  L'h  S-D  message.  On  the  average,  a  random  ACK  arriving  at  Qh  sees 
(Z.-D/2  ACKs  in  front  of  it  and  (L- 1)/2  ACKs  arrive  after  it  does.  Each  inter-arrival  is,  on 
the  average,  1/S,  units  of  time,  i.e.. 


which  is  Eq.  (4.31 ). 


On  the  other  hand,  as  Sh 


°°,  we  have  pb 

FIT*!  -  -  - 

a 


—  0,  a  —  Sh  and 


J 

-V, 


(4.32) 


Again,  this  is  intuitively  clear  since  as  Sh  —  no  special  control  message  needs  to  be 
generated.  Each  ACK  is  picked  up  by  a  D—S  message,  which  arrives  each  \/Sh  units  of  time 
after  our  ACK  joins  the  queue  (the  D-S  message  generation  process  is  assumed  to  be  Pois¬ 
son). 


The  effective  traffic  rate  is 


•**  =  -S  ,  4  .S',  P,  , 

The  first  term  is  of  course  the  D-S  message  rate,  while  the  second  term  is  the  extra  traffic  gen¬ 
erated  because  the  l.  limit  is  reached  (i.e.,  the  ACK  for  S-D  message  finds  L-l  ACKs  in  Qb) 

(I  -ptW-'/d-pf) 

Therefore 


As  .S'/,— < 0.  p*— I,  and  V 
sage;  as  ,S^—  «>,  .S'N,—  .S'„. 
all  as  .S'ft—<». 


V*  =  SA  +  .S', 


(1 


1- 


(4.33) 


—S',/ L.  That  is  every  L  S-D  messages  contribute  an  extra  D-S  mes- 
This  is  again  true  since  no  special  message  needs  to  be  generated  at 


Now  a  computer  communication  network  can  be  described  as  if  it  were  a  network  of 
queues  with  external  traffic  .S/,,'s. 

The  one-way  message  delays  can  again  be  obtained  from  L.  Klcinrock’s  equation  IKLE1 
7M.  The  round-trip  delay  (message  delay  plus  ACK  delay)  of  a  particular  flow.  e  g..  S-D  flow, 
is 
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Ts-d  +  Tb  +  Tp-s 

where  Ts-d  ( TDS )  is  the  message  delay  in  the  network,  and  Tb  is  the  waiting  time  in  Qb. 

In  Fig.  4.12  we  show  the  round  trip  delay  T,  (normalized  with  respect  to  the  message 
length  x)  of  logical  channel  5-4  when  we  apply  the  piggy-back  scheme  to  our  little  network  in 
Fig.  4.2.  In  the  figure  p*  is  the  utilization  factor  of  the  most  heavily  loaded  physical  channel 
when  L  =  1  *  (e  g.,  physical  channels  5-3,  3-2,  and  2-4).  We  increase  traffic  uniformly  so  that 
p*  increases  from  0.16  to  1.28.  The  most  we  can  do  is  to  increase  traffic  to  the  extent  that 
p*  <  2  since  at  this  point,  with  L=°°,  the  effective  traffic  (in  this  case  no  ACKs  contribute  to 
the  effective  traffic)  just  loads  up  the  most  heavily  loaded  channels  (i.e.,  the  channel  utilizations 
are  one) . 

Observe  that  when  p*  is  small,  ACKs  for  one  flow  are  seldom  picked  up  by  messages 
of  the  opposite  flow  (since  the  traffic  is  low).  Therefore,  if  L  is  too  large,  then  so  is  Tb.  As 
traffic  increases,  p*  increases  and  we  have  more  intensive  traffic  to  pick  up  ACKs.  It  begins  to 
be  beneficial  to  have  L  >1.  In  our  little  network’s  case,  we  see  this  occurs  around  p*  *=  0.64. 
Thus,  as  p*  increases  we  want  L  to  increase  so  as  to  make  the  round  trip  delay  small.  Of 
course,  as  p*  increases  more  the  network  starts  to  be  saturated  (as  shown  by  the  curve 
corresponding  to  p’  =  1.28). 

4.3.2  Piggy-back  Scheme  with  Limit  on  Number  of  Accumulated  ACKs  and  the  Length  of 
ACK-waiting  Time 

Let  us  again  consider  the  network  in  Fig.  4,9.  We  have  everything  remaining  the  same 
except  that  we  add  one  more  condition:  namely,  that  if  the  first  ACK  in  Qb  has  been  there  for 
Ta  units  of  time,  then  a  special  control  message  is  generated  to  pick  up  all  the  ACKs  in  Qb. 
Remember  that  this  is  to  prevent  the  indefinite  waiting  of  ACKs  in  Qb. 

To  describe  Qb ,  no  state  diagram  such  as  that  in  Fig.  4.10  can  be  drawn  because  of  the 
Ta  limit.  Let  us  refer  to  the  time  diagram  in  Fig.  4.13. 

As  time  goes  on,  Qb  undergoes  repeated  cycles,  each  of  which  consists  of  an  empty 
period  followed  by  a  filling  period.  An  empty  period  is  ended  by  the  first  arriving  ACK  (for  a 
S—D  message)  which  starts  the  filling  period.  The  filling  period  is  ended  when  a  D—S  message 
is  accepted  (by  the  network),  when  the  Ta  time  limit  expires,  or  when  L  ACKs  are  collected  in 

Qb- 


Refer  to  Fig.  4.14(a);  let  Psb  k  Pr[a  filling  period  is  ended  by  a  D-S  message)  =  Prla 
D—S  message  is  accepted  into  the  network  before  (L-l)  ACKs  for  S—D  messages  arrive  in 
node  D  in  the  time  period  (0,  Ta )  after  a  filling  period  starts).  Note  that  we  only  need  (L-l) 
ACKs  to  end  a  filling  period  because  the  period  is  started  by  an  ACK  at  time  zero.  Let  A  be 
the  event  that  fewer  than  (L-l)  S-D  messages  arrive  at  D  in  (0,r);  and  let  B  be  the  event 
that  the  first  D-S  message  is  accepted  into  the  network  at  t  +  dt.  Then 
Ta 

Psb  =  f  Pr\A]-Pr[B]dt 


*  Remember  that  when  L=l,  each  ACK  is  sent  back  to  the  source  as  an  ordinary  message. 
These  messages  are  also  included  in  calculating  p*. 
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V'r. 


•  EMPTY-*] 

- - FILLING - * 

-—EMPTY—* 

—FILLING— 

»ST  ACK  GROUP  1st  ACK  GROUP 

ARRIVES  AT  ACKs  ARRIVES  AT  ACKs 

0,5  OUTPUT  Q  OUTPUT 


Figure  4.13.  Time  diagram  of  Qb  activity. 


■  /  I  9H'!~  e~a' dt  where  “  -  s/ + 


0  K- 0  “ 


L-2 


where 


-  I  (l-PtWMO.r,) 

k- o 


“•  -  srh 

/*:  (0, 7% )  is  defined  in  Appendix  A 


(4.34) 


Refer  to  Figure  4.14(b).  Let  PL  £  Pr[a  filling  period  is  ended  because  L  ACKs  are 
collected  in  Qb],  Then  let  <7  be  the  event  that  no  D-S  message  is  accepted  in  (0, r ) ;  let  H  be 
the  event  that  (L  — 2)  ACKs  for  S—D  message  arrive  at  Qb  in  (O.r);  and  let  F  be  that  one  ACK 
for  an  S—D  message  arrives  in  (r,  r+dr).  Then, 


PL  -/  Pr[G\ PrlHlPrlF]  dt 


-  J  <’~V- 


(S,r) 


L-2 


-S,i 


(L  —2). 


Sf  dt 


O.Ta)  (4.35) 

And  finally  refer  to  Fig.  4.14(c);  let  PT>  ±  Pr[a  filling  period  is  ended  because  the  first  ACK  in 
Qb  has  waited  Ta  units  of  time].  Also  let  X  be  the  event  that  no  D-S  message  is  accepted  in 
(0.  T„),  and  Y  be  that  less  than  (Z.-1)  ACKs  for  S-D  messages  arrive  at  Qb  in  (0.  Tal. 
Then, 
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Figure  4.14.  Three  types  of  filling  periods. 
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pT  -  mi  mi 

a 


-v, r  t2  -s,r 

L*  is  i 

X-0  *  • 

S/A  -ar„ 


a-o  a 


a:i 


(-2  {aT„)K  _ar„ 

=  L  p*  Y'  e 


a-  o 


Observe  that  as  T„  —  °°, 

^  -  I-p/  -' 

PL-i>l~' 

PTo  -  0; 

We  have  (as  it  should  be) 

Ps  +  PL  +  PT  =1 

as  ra— 0, 

^  -  0 


Again, 

>\  +  pl  +  pt,=  i 

In  any  case,  regardless  of  the  value  of  fa,  the  quantity 

P  4  Pv,  +  Pt  +  Pra 

must  equal  to  1.  Indeed,  from  (4.34),  (4.35),  (4.36)  we  confirm  this  as  follows: 
P  -  /V  +  PL  +  PT 


'a-o  i  ,_o  /• 


r  1  I  * 

>-<•  •+  X  P»*  I-X 

A  — 1 


*  ( aTJ 1 


”  I' 


-pM-I 


(«Ta) 


<*’  ~aTa\ 


"o 


ri-l  A" 

-  I  Pf  '"I 

lA ‘—I  i-C 


A  (or.)1 


“  !■ 


"r-|  I  +  X  pf | 

A'-l 


(aTa)K 


K'\ 


"r’l  1 


(4.36) 

(4.37a) 

(4.37b) 

(4.37c) 

(4.38a) 

(4.38b) 

(4.38c) 
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(<*Ta)K 

A'! 


(aT,)*  -ara 

-  n  « 

a:i 


-  i_,.~ar-  +  1 

“  P*  (/.-!)! 


e~"r“  +  ?-“r. 


-  Pb 


L-l 


(aTa)L~'  -aTa 
(L-l)!  *' 


=  1 


(4.39) 


We  now  calculate  the  expected  value  of  the  filling  period  Tf . 

Let  Ta  4  £17",  I  rj  4  expected  value  of  Tt  given  the  filling  period  is  ended  because 
the  Ta  limit  is  reached.  Of  course, 

Ta  =  Ta  (4.40) 

Let  Ts  4  E[T,  |,S’J  4  expected  value  of  T,  given  that  the  filling  period  is  ended  by  an  D— S 
message.  Then, 

r.  ,  **| 

W"'  - - 


y!  }  Ml  Ml  f-.,A 

*•- 1  „»*'  a  (K+l>! 


A  -0  0  « 


1-2  A' 41 

X  A±L(1_P  )p*-/  (0,ro) 

A— 0  « 

Ps. 


(4.41) 


And  let  TL  4  E\T,\L\  4  expected  value  of  Tt  given  the  filling  period  ended  because  L 
ACKs  accumulated  in  Qh. 

r 


h 


/ 


S,  7 


(S,i)L-2e  ' 


(£ 


Z2)i - S' 


£-1 


Pa  *  /,.-.<0,ra) 


(4.42) 


From  (4.40),  (4.41),  (4.42) 
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T,  =  t:\T,  |.s;|- 7\„  +  El T,  I L\P,  +  E\T,\Ta\  Pra 

1.-2  1,1 

-  x  /A+,(0 ,r„) 

~0  « 


L-l 


+  —  pjM  +  r„  I  P*fc 


X-0 


(a7-/ 

K' 


-<*T„ 


Observe  that  as  7^  —  °°  ,  we  have 


i-2 


r,-Z  — (i-p*w  + 


*-0 

1-Pa 


Pa 


(1  +  2p*  +  3p;  +  ■  +  (T~l)p£  2)  +  ~ — -p£  1 


-  -  <1  +pa  +Pa2  + 


+  P^-J)  - 


i 

«  !~Pa 


And.  as  Tu  —  0 

r,  -0 


(4.43) 


(4.44) 


The  effective  traffic  .S'*  ( S>,  +  the  special  control  messages  generated  when  L  or  Ta 
limits  are  reached)  is  derived  as  follows:  Consider  V  (a  large  number)  cycles  (remember  each 
cycle  consists  of  an  empty  period  followed  by  a  filling  period).  Among  N  cycles  only 
( PT  +PL  )  V  of  them  will  generate  extra  messages. 


Therefore,  for  large  V, 

•V  -  s„  + 


V  (Pr  +  PL) 


/VV(7i  +  --)  +  />vV<  Ts,  +  —)  +  PTq  H  ( T a  +  —) 


(4.45) 


where  the  denominator  of  the  second  term  is  the  average  time  length  of  the  N  cycles.  Thus, 

Pr,  +  PL 


V  =  sh  + 


(PL  Tl  +  P.TSk  +  PT  Ta)  +  ~~ 

A  A  u  5, 


=  sh  + 


Pt  +  P,. 


T,  + 


S, 


(4.46) 


We  observe  that  as  ,  from  Equations  (4.37),  (4.44) 

1 


Sa«,  =  Sh  + 


I-P/"1  +  1 


ad-p*)  S', 


(paL-’+0) 


(4.47) 


=  5,  + 


Equation  (4.47)  is  exactly  the  same  as  Equation  (4.33)  (where  we  had  no  limit  on  time)  as  it 
should  be.  On  the  other  hand,  if  TA  —  0,  then 

S*,-  S»  +  (1+0)  /  [0+1/  S,]  =  Sb  +  S, 


This  is  correct  since  each  ACK  for  a  D—S  message  now  generates  a  special  control 
message. 


Having  done  the  above  calculations,  we  are  ready  to  derive  the  average  delay  time  Tb 
for  an  ACK  to  stay  in  Qh  Refer  to  Fig.  4.14a.  Let  />[A,/|Sa]  k  Prlthe  filling  period  lasts  t 
units  of  time  and  ends  with  K+ 1  ACKs  in  Qh  I  it  is  ended  by  a  D-S  message]  = 
P[K  j,Sb\l P, ,  where  ”1"  is  the  first  ACK  which  starts  the  filling  period,  and 

b 


P[K,i\Sh) 


s.  i 


sh  dt 


iSft)K  e 
K\ 


for  0^K^L-2.  r<  Ta 


for  K^L-\  or  i^Ta 


(4.48) 


Let  P[K,t\L]  k  Prlthe  filling  period  lasts  /  units  of  time  and  ends  with  K+ 1  ACKs  in  Qh  I  it 
ends  because  the  Z.  —limit  is  reached]  =  P\K  j ,L\! Pi 


P[Kj\L] 


-V 


-S,i 


e  -  ■  ■  e  '  S,  dt  .  for  K  —  L- 1,  ,<  Ta 

0,  otherwise 


(4.49) 


Let  P[K\Ta\  4  Prfthe  filling  period  ends  with  A'  +  l  ACKs  in  Qh  given  that  it  ends 
because  the  Ta  limit  is  reached]  =  P\K ,Ta\/ PTa 


P\K\Ta\ 


K\ 


e~SJ-  /  Pt 


0,  for  /<  Ta  or  A>L-1 


for  0$  A ^L-2,  t~Ta 


(4.50) 


Before  proceeding  further,  we  make  an  observation  (which  is  easy  to  prove): 

If  we  know  that  during  a  time  period  [0,t],  we  have  k  Poisson  arrivals,  and  at  i  they 
are  all  served  together,  then  the  average  waiting  time  is 

Now,  to  calculate  Tb,  the  average  waiting  lime  in  Qb,  consider  a  large  number  A  of 
filling  periods.  Define  Nsb{K.t)  k  the  number  of  filling  periods  ended  by  a  D-S  message  at 
(t,r+dr),  with  K  +1  ACKs  in  Qh  (we  call  this  type  of  filling  period  the  (K,t,Sh)- type).  Then. 

N^K.i)  =  N PshP[Kj\S„)  (4.51) 

Let 

i<h(K.i)  -  NJKj)\K  j-  +  r]  (4.52) 
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Then  i,  is  the  total  time  each  ( A\/,  S'*) -type  ACK  filling  periods  wait  in  Qb.  Note  that 
the  second  term.  /,  in  Eq.  (4.52)  is  the  time  that  the  first  arrival  (who  starts  the  filling  period) 
waits  in  Qh. 


Similarly,  STo(K)  A  the  number  of  filling  periods  that  end  because  the  Ta  limit  is 
reached  and  K  ACKs  have  collected  in  Qh  ((K,Ta  )  type). 


Ar  (A)  =  N  PT  P[K\Ta\  (4.53) 

'a  a 

Let  ta(k)  A  total  amount  of  time  the  ACKs  of  (X\7"0)-type  filling  periods  wait  in  Qh  collec¬ 
tively.  Then, 


ta(K)  =  NTa(K)- 


(4.54) 


Similarly,  let  \L(i)  A  the  number  of  filling  periods  that  end  at  /  because  the  /.-limit 
is  reached.  Then, 

Nl(i)  =  MPlP[i\L] 


/,.(/)  =  Nl(i)  \i  +  0  +  (L- 2)  y  ] 


(4.55) 


The  first  term,  i,  is  the  length  of  time  that  the  first  ACK  waits;  the  second  term,  0,  is  the 
length  of  time  that  the  L'1'  ACK  waits.  Thus. 

t  -2  L  -2 

X  J  ish(K.t)dt  +  £  ta(K)  +  f  tL(t)dt 


Th  = 


(4.56) 


/"2r°  L  -2  u 

£ "  /(K  +  D/V  (K.i)di  +  £  Nt  (K)  (K  +  \)  +fNL(t)Ldr 

*.  -0 1  0  A  .nl  u  n 


L -2 

I 

A-0 


The  denominator  is  the  total  number  of  ACKs  gathered  in  the  A  filling  periods  and  the 
numerator  is  the  total  amount  of  time  that  ail  the  ACKs  stay  in  Qh  .  Tb  can  be  rewritten  as 


,  A  i+.4  ->+A} 

Th  A  — - - - - 

B  \  + 


(4.57) 


with  each  term  corresponding  to  that  of  the  last  equation. 


We  have,  after  a  little  calculation 
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A  -0  A 


(4.58) 

(4.59) 

(4.60) 

(4.61) 


57 


r 


4 ,  =  .v  p/-1  /t_,(o.rtt) 


and 


=  V  Lfif, 

We  again  use  the  mathematics  in  Appendix  A  to  derive  the  above  equations. 


Let  us  now  check  the  limiting  value  of  Th.  When  Ta  —  °o, 
.  *  (A.+1XA+2)  ,  L(L- 1) 

£  " - £ - +  <■* 

f  h  * 


r-i 


£  (K  +  IXl-p^+Lp*4 

A  -0 

I2+T  pjN<K  +  l)<K+2)-K(A:+l)l 

2“  1  x-i 


1  +  p/,  +  p/  +  •  ■  •  +  p/  2  -  (L- l)p*‘  +  Lfi‘h 


.  1  - 1 


(l+2p„+3p/+  ■  +(/.  —  1  )p/  *) 

ad— p /•)/(!— p*) 


d+p/,+p/+  ■  ■  -Xp /  2—  ( /L  —  1  )p ^  1  )/<  1  — Pa ) 

a(l— p/)/(l— p/,) 


"P* 


» — Pa 


-  (L  — 1  Ip/--1 


ad-p/l 


This  is  exactly  (4.30)  where  we  have  no  wailing  time  limit! 


From  Appendix  A,  when  Ta  —  0, 

.  Jk  <«T„)'  _„r  [  (aTa)2 

Ik  (0,  Ta )  =  1  — a—  <•  u  —  I  -  l-n70-l - r2 —  + 

,.(}/■  1  * 


(«TU2) 


1  +a  r + 


Therefore, 


(4.62) 

(4.63) 


(4.64) 

) 

(4.65) 
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£  <l-p/.)/j*‘U'  +  l><A+2)<u/;;l  2,« -*•  <../,» 


,  ,  i  «/;,!• 

i  <*  / ,  -  -  + 


L  ( 1  rdJ  /.-i 


p/. 


X  + 1 1 1  1— pyTp*  -  y" 


—a  /., + 


<«  /;,)■ 
2 


—  («  /„  —  (o  /,j )~  +  .  .  I 
or 


<«  7', ) 

-  „  r,  +  — f 


0  linearly  wiih  /],  as  expected. 


Figures  4.15  and  4.16  show  ihe  results  (  /),  and  round  trip  delay  of  logical  channel  5-4) 
of  applying  the  scheme  lo  our  little  network  (Figure  4.2) 

Note  that  when  the  externally  applied  load  is  small  (e  g..  p‘=0.l6)  we  want  to  keep  L 
and  Ta  small.  The  reason  again  is  that  when  traffic  is  low.  it  does  not  pay  to  wait  at  Qh.  On 
the  other  hand,  when  the  applied  load  is  high  (e  g..  p  '=<).%  ).  there  exists  an  optimal  (Z..  Tu  ) 
value  to  minimize  7,  In  Figure  4  P.  the  minimal  round  trip  delay  T,  is  plotted  (obtained  by 
adjusting  /.  and  7„  )  versus  p  '  Observe  that  as  /»’  increases  front  zero.  T,  increases  linearly 
with  p*  at  iirsit  then  the  piggy -back  scheme  begins  to  become  effective  in  reducing  T,  as  p * 
becomes  medium  large  (0.75  to  1.25).  finally  the  network  is  saturated  as  ft*  becomes  too  large 
(>  1.25).  The  above  derivation  provides  a  method  (static  policy)  for  a  network  to  dynamically 
adjust  the  number  of  ACKs  to  be  piggy -backed  together  and/or  the  time  length  of  delayed 
piggy-backs. 

4.4  The  Time  Out  Scheme 

In  the  real  world,  transmission  channels  are  always  noisy  and  hardware  and  software  at 
the  nodes  are  error  prone.  In  a  computer  communication  network,  data  may  become  lost  (e  g., 
because  of  uncorrectable  errors  in  the  heavier)  along  the  path  from  its  source  to  its  destination. 
Therefore  a  time-out  scheme  is  usually  needed  to  automatically  retransmit  those  messages 
whose  ACKs  do  not  come  back  to  the  source  within  a  specified  period  of  time  and  are  thus 
presumed  lost. 

In  this  section,  we  try  to  incorporate  the  time-out  scheme  into  our  llow  control  model 
The  following  assumptions  are  made 

(1)  Newly  arrived  messages  are  accepted  or  rciected  as  before,  re  .  accepted  only  if  there 
are  permits 

(2)  The  probability  that  a  message  get>  lost  at  hop  is  /’ 

(.7)  When  the  AC  k  of  a  message  does  not  come  hack  to  the  source  of  the  message  within 

the  time-out  limn  /„  since  the  instant  ol  the  last  transmission  of  that  message,  the 
source  will  immediately  retransmit  ii 


XU 


Figure  4.15.  Delay  of  logical  channel  5-4  a s  function  of  L  and  T  at  p*  — 0.16. 


(4)  The  source  will  slop  retransmitting  a  message  once  it  receives  an  ACK  for  that  message 
whether  the  ACK  is  or  is  not  for  the  most  recent  transmission  of  that  message.  The 
subsequent  ACks  (if  any)  for  the  same  message  are  just  ignored. 

(5)  An  ACK  is  treated  here  as  an  ordinary  message  except  that  it  does  not  consume  a  per¬ 
mit. 

<fe)  Each  retransmission  of  a  message  uses  up  one  permit.  If  there  are  no  permits  the 
retransmission  will  use  a  "credit  card"  i.e..  it  will  consume  a  permit  beforehand,  the 
next  generated  permit  then  makes  up  this  deficit.  Thus,  a  node  will  destroy  the  next  n 
generated  permits  if  there  are  n  permits  consumed  in  advance  by  the  retransmitted 
messages. 

(7)  The  destination  acknowledges  the  correct  messages  only.  And  it  accepts  only  the  first 
correct  copy  of  a  message 

(8)  For  simplicity  we  make  the  approximation  that  the  round  trip  delay  of  a  message  (mes¬ 

sage  delay  +  ACK  delay)  never  exceeds  3  T(]  if  the  message  does  not  get  lost.  This  is 
a  reasonable  assumption  since  70  is  usually  set  to  be  greater  than  the  average  round  trip 
delay  of  messages.  Notice  that  for  an  exponential  distribution  with  average  it.  the  dis¬ 
tribution  function  /]3<r]=f  —  t/v  =  1— e'3  =  0.95.  Therefore.  Prlround 

Ju  (T 

trip  delay  >  3<r|  =  0.05.  And  7n  is  usually  »<r. 

(9)  For  easy  mathematical  manipulation,  we  assume  that  for  each  logical  channel,  the 
external  input  traffic  plus  the  retransmitted  traffic  is  always  larger  than  the  permit  gen¬ 
eration  rate  .c,  < this  implies  that  the  total  traffic  of  logical  channel  r  that  enters  ihe  net¬ 
work  can  be  approximated  by  ,g, .  i.e..  the  heavy  traffic  situation). 

Under  the  above  assumptions  we  now  analyze  our  time-out  scheme.  We  first  calculate 
the  average  number  of  transmissions  V,  of  a  message  Then  the  throughput  of  a  logical  chan¬ 
nel  r  is 

V,  =  ~  (4.66) 

'  I 

Next,  we  derive  the  average  delay  D,  of  a  message,  as  the  source  of  the  message  sees  it 
(i.e.,  the  time  period  between  the  instant  of  the  first  transmission  and  the  instant  of  receival  of 
the  first  ACK  of  a  message).  Let  P,  k  Prlthe  round  trip  time  of  a  particular  transmission  lies 
in  the  interval  I ( /—  1 )  Fn./Tol  given  the  transmission  is  not  lost],  i  =  1.  2,  3. 

Note  that  by  assumption  (8).  we  have 

/Ji  +  +  Py  =  I 

Define  P„.  k  Prla  transmission  results  in  retransmission]  and  Pi  k  Pr la  transmission 
gets  lost].  We  have 

P,  =  !  -  fl  (1  -  r:  >  <4. 67) 

.«  /O  > 

where  Hr)  =  {hop  /Ihop  /  is  in  the  round  trip  path  of  logical  channel  r) 
and 
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f‘r,  =  P,  +  PM-P  |> 


(4.68) 


where  X  k  1- V 


Figure  4.18.  The  event  time  diagram  for  P[X,=n J. 


Now  refer  to  Figure  4.18.  We  have 

m=«]  =  pr'K  +  pr2P,p 2  +  pr'PrPyK  +  pr2-p,-pyp„  <4.f>9> 

The  firs!  term  means  that  the  only  'not-lost'  transmission  is  the  one.  and  whose 
ACK  comes  back  within  the  time-out  limit  7n,  i.e.,  within  The  second  term  says  that 

the  first  transmission  not  lost  is  the  (//-l)'"  one,  whose  ACK  comes  back  within  2  T0,  i.e.,  in 
.  Thus  the  //"’  transmission  must  be  made  (but  we  do  not  care  about  the  result  of  this 
transmission).  The  third  term  represents  the  case  where  the  first  transmission  not  lost  is  the 
(//  —  l)"’  one,  and  its  ACK  comes  back  within  3  Tfl  (i.e..  in  (r„  +  |./„„2>),  and  the  ACK  of  the  n'" 
transmission  comes  back  within  Tn.  The  last  term  tells  us  that  the  first  transmission  not  lost  is 
the  i.n-2)  one,  but  its  ACK  comes  back  within  3  Tn.  and  the  ( // —  1 ) '"  transmission  results  in 
retransmission.  We  have  no  more  terms  because  of  assumption  (8).  Since  we  have  fixed  rout¬ 
ing  and  the  messages  are  first-come-first-served.  if  the  ACKs  are  not  lost  they  are  never 
returned  out  of  sequence;  therefore,  the  third  term  is  impossible,  and  is  dropped.  Thus,  Equa¬ 
tion  (4.69)  can  be  rewritten  for  later  use  as 

=  Pr'  P„+PT2  P  P:+P:  '  P  PyP ,, 

±  P„\  +  P„2  +  P„y  (4.70) 

with  P,„  corresponding  to  the  /"'  term.  The  average  value  of  V,  is 

P-,+P\P  P 

£[/V,l  =  £  «  P l  V,  =  (,  1  =  P2  +  2 ^  (4.71) 
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Let  us  now  try  to  find  the  round  trip  delay  D,  as  the  source  sees  it,  i.e.,  the  time  period 
between  the  instances  of  the  1st  transmission  of  the  message  and  the  reception  of  the  1st  ACK 
for  that  message. 

Let  E[D,\P„,]  represent  the  average  of  Dr  given  Pnj  =  \,j  -  1,  2,  or  3.  Let  T , 
denote  the  time  period  between  ( / — 1 )  T0  and  the  time  that  the  ACK  for  a  particular  transmis¬ 
sion  comes  back  to  the  source  given  that  it  comes  back  in  ( ( /  —  1 )  7o,/T0)  i  =  1,  2,  3.  And 
finally  let  T,  denote  the  round  trip  delay  of  a  particular  transmission  i.e.,  the  time  period 
between  the  instant  a  transmission  finishes  and  the  instant  the  ACK  for  that  particular 
transmission  returns.  Observe  the  difference  between  T,  and  Dr  .  T,  can  be  °°  if  that  particu¬ 
lar  transmission  gets  lost. 

Referring  to  the  time  diagram  in  Figure  4.18,  we  have 
E[D,\Pj  =  (/»— 1)  T0  +  fir,] 

E{Dr\Pj  =  ( /i-l )  T0  +  £[min  {7*2, r,|I 

E[Dr\Pj  =  («  — 1)  7o  +  £[min  \Th\T2\Pre],  T,\] 

The  same  assumption  that  we  have  fixed  routing  and  messages  are  first-come-firsl- 
served  (hence  messages  (and  ACKs)  never  arrive  out  of  sequence)  enable  us  to  simplify  the 
above  equations  to 


E[Dr\Pj  =  <//—!)  7o  -F  E[T\) 

(4.72) 

E\D,\PJ  =  (//-I )  To  +  £(721 

(4.73) 

ElDt\P„, ,  =  ( //—\ )  T0  +  E[T} ) 

(4.74) 

£[0,1  =  £  E[Dr\i „)  P„, 

n-  1 

(4.75) 

With  Eq.  (4.66)  for  throughput,  (4.71)  for  average  number  of  transmissions,  and 
(4.75)  for  the  average  message  round-trip  delay,  we  have  solved  the  system.  The  only  thing 
left  is  to  derive  £ [  7'i  1 ,  £(  7~2l .  and  E[T})  . 

This  can  be  done  as  follows:  We  first  define  the  following  quantities.  X,  ^  the  sum  of 
all  traffic  which  is  routed  through  physical  channel  /.  By  the  heavy  traffic  assumption 

A,  =  I  *,  (4.76) 

rt  Rl,) 

where 

/?(/)  =  {logical  channel  r  I  message  of  logical  channel  r  are  routed  through  physical  channel 

-I. 


Let  T„  k  the  round  trip  delay  of  a  message  given  Ph  =  0,  and  there  is  no  time-out 

scheme. 
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We  have  from  J.  Wong's  equation 


km  -  n 

,i  lb) 


pC  ,—K, 

S  +  (fiC-K,) 


(4.77) 


for  a  particular  logical  channel  r .  From  this,  the  probability  density  fT ■  it)  can  be  obtained  by 
inverting  T‘,(s)  Thus,  for  i  =  1,  2,  3 

E\T,  1=  J  /  lT  (r|/€l(/— l)7'0,/r0l)  di  (4.78) 

(,-nr0 


We  apply  the  above  time-out  scheme  to  our  little  network  again.  The  results  are  shown 
in  Figures  4.19  and  4.20. 

Fig.  4.19  shows  the  throughput  and  delay  of  logical  channel  5-4  in  the  little  network. 
Observe  the  sharp  decrease  of  throughput  and  sharp  increase  in  delay  at  heavy  external  load. 

The  long  delay  is  because  of  the  nearly  saturated  network  capacity  while  the  decrease  in 
throughput  is  because  of  the  excessive  retransmission  of  messages  which  is  caused  by  the 
time-out  scheme.  Note  that  the  curves  corresponding  to  To  =  0.5  T„  ( T„  is  the  average 
round-trip  delay  of  logical  channel  5-4  when  P,,=  0  and  there  is  no  time  out  mechanism)  are  not 
accurate  since  assumption  (71  no  longer  holds.  The  dashed  line  for  the  throughput  shows  the 
true  behavior. 

Fig.  4.20  shows  the  "power"  (defined  below)  versus  TJTrt  for  different  channel  error 
rates  Ph's.  The  optimal  (maximum  power)  time-out  line  is  also  shown  as  the  broken  curve. 
The  power  for  a  logical  channel  is  defined  as 

(  throughput  with  time-out  scheme  I 

p,mvr  =  ( throughput  without  time-out  scheme J  =  fAA’, )  79) 

round  trip  delay  with  time-out  scheme  and  error  ( D,/T„ ) 

round  trip  delay  without  time-out  scheme  and  error 


This  is  a  measure  of  the  effectiveness  of  the  time-out  scheme  under  a  set  of  network 
parameters.  The  maximal  value  of  power  is  1. 

4.5  Buffered  Input  Model  and  Inter-netting 

Networks  are  usually  connected  by  note  wavs  where  differences  in  protocols  are  resolved, 
and  where  billing  and  flow  control  between  networks  takes  place. 

In  this  section  we  discuss  how  to  apply  the  idea  of  flow  control  based  on  limiting  the 
permit  generation  rate  to  the  inter-netting  case. 

Consider  two  networks,  V,  and  V2.  connected  by  a  gateway  O'  as  illustrated  in  Fig. 
4.21(a).  Nodes  a.  h.  c,  communicate  with  nodes  /.  /.  A,  ■  •  by  first  sending  their 
messages  to  the  gateway  (>.  After  processing  these  messages  the  gateway  G  will  send  them  to 
their  destination  /.  /.  or  A.  etc. 


Or(Vr) 

Figure  4.19.  Throughput  and  delay  as  a  function  of  time-oul  length  as  P,  =  10“ 1 


Figure  4.20.  The  optimal  ti 


For  simplicity  let  us  assume  that  all  Hows  from  .V j  to  /V2  are  controlled  by  the  permit 
queue  qp  and  a  permit  generation  rate  The  procedure  used  is  similar  to  that  used  in  Sec. 

4.2  except  that,  contrary  to  Sec.  4.2,  when  an  \',-/V2  message  finds  no  permit  in  queue  qp,  it  is 
not  rejected.  Instead,  it  will  be  queued  in  q\  to  wait  for  a  permit  to  be  generated.  The  rea¬ 
son  messages  are  not  rejected  at  G  is  to  assure  the  users  that  once  the  network  accepts  their 
messages  it  will  do  its  best  to  deliver  the  messages  to  the  destination.  Also,  for  the  sake  of  net¬ 
work  utilization,  it  is  not  a  good  thing  to  throw  away  messages  in  which  we  have  invested  so 
much  of  the  resources  (by  sending  them  from  their  sources  to  the  gateway  G ). 


n2 


(a) 


Figure  4.21.  Flow  control  between  networks. 


The  above  discussion  leads  us  to  investigate  the  following  scheme. 

4.5.1  The  Buffered  Input  Scheme 

Consider  the  control  scheme  shown  in  Fig.  4.21(b).  Assume  that  V,  to  V2  messages 
arrive  at  q,„  at  a  rate  y.  They  are  accepted  by  network  Yj  only  if  they  can  secure  permits  from 
qn,  which  is  a  finite  buffer  of  size  V,  and  which  accommodates  permits  generated  at  a  rate  g. 

The  system  can  be  described  by  a  Markov  Chain  as  shown  in  Fig.  4.22(a).  In  the 
diagram,  a  state  (/./')  means  that  there  arc  /  messages  in  q,„,  and  /  permits  in  qn.  Observe  that 
it  is  impossible  for  ;^0  and  0  simultaneously;  and  that  /  is  always  less  or  equal  to  V.  Thus, 
a  state  is  either  0,0).  or  (0,/)  with  /$  V. 
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<b> 

Figure  4.22.  The  state  diagrams  of  qN|S,  and  qr. 
By  transforming 

(0,/)  —  (  Y-/> 
and 

(/.  0)  —  (\  +  /> 

the  diagram  in  Fig.  4.22(a)  becomes  Figure  4.22(b). 

This  is  just  the  state  diagram  of  an  M/M/1  queue!  Therefore. 

p.  =  ( |  _  £)<  L>A.  =  <!-„),/ 

Si  K 

where  p  4 

After  transforming  back  to  the  original  system,  we  obtain 

P«,  =  P\-,  -  <l-p)p'  .  /  =0.  1.  •  .  V 

and 

P,o  =  P\ 4,  -  d-p)pw'  .  #  —  1,  2.  -  •  - 

Thus,  the  average  number  of  messages  in  q,„  ts 
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9 


t 


^-X'^-u-p^X'p' 


,V+l 


(1-p) 

The  average  delay  in  Qm  is 


D  „  ll/ilii 

f 

I  -p 


/V  +  l 


g-y 


(4.83) 


(4.84) 


Observe  that  the  inter-departure  process  from  qm  is  not  Poisson.  This  can  be  seen  as 

follows. 

Case  1:  The  departure  message  sees  qm  empty  and  qp  not  empty;  then  the  Laplace  transform  of 
the  probability  distribution  of  the  interdeparture  time  is 


/4*(s  |  case  1)  -  -  — 
s  +  y 

(4.85) 

Pi  case  U  =  X  f’o,  «  (I  ~p)  X  PN~'  =  P~PN 

i-\ 

(4.86) 

Case  2:  The  departure  message  sees  Qm  not  empty  and  Q„  empty. 

A*(s  I  case  2)  -  g 
s  +  g 

(4.87) 

Please  21  *  £  P,o  =  (1  -plp^X  P'  =  PN*' 

,-i  -i 

(4.88) 

Case  3:  The  departure  sees  both  Qm  and  Qp  empty. 

y4  *( s  I  case  3)  «  g  g  + — J — 

y  +  g  s  +  g  y  +  g  s  +  y 

(4.89) 

Please  31  =  Poo  =  (l-p)p  v 

(4.90) 

Therefore, 

3 

A*ls)  =  X  ^‘(slcase  /)  Please  /] 

.-i 

N(s) 

( s  +y )  ( s  +£ )  (  s  +y  +g) 

(4.91) 

where  /V(s)  is  a  polynomial. 


We  see  that  4*(s)  cannot  be  Poisson,  since  if  it  were,  the  A ’(s)  must  cancel  the 
denominator  in  such  a  way  that  only  S+y,  S+g,  or  S+y+je  remains. 

This  implies  that  if  the  inter-departure  process  is  Poisson,  then  it  is  Poisson  with  a 
parameter  y,  g,  or  y+g.  None  of  these  can  be  the  case  for  our  system  and  therefore  it  is  not 
Poisson. 


t 
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We  see  that  we  can  apply  the  above  model  to  the  flow  control  at  gateway  G  in  Figure 
4.21(a).  The  entire  system  can  be  analyzed  by  separately  analyzing  S,  and  N2.  The  only  con¬ 
venient  assumption  we  have  to  make  is  that  the  traffic  which  enters  N2  from  .V ,  is  Poisson  (and 
which  we  have  just  shown  is  not  true). 

Since  it  does  not  generate  any  more  insight  to  analyze  N\  and  N2  in  detail,  we  shall 
leave  this  section  here. 

4.6.  Conclusion 

We  have  presented  two  flow  control  schemes,  each  based  on  limiting  the  permit  genera¬ 
tion  rates  for  logical  channels. 

The  advantages  of  these  kinds  of  flow  control  schemes  are: 

(1)  They  require  no  end-to-end  acknowledgements.  We  have  shown  that  our  schemes  pro¬ 
vide  good  network  performance  most  of  the  time.  Only  at  infrequent  instants,  when 
the  traffic  pattern  and/or  the  service  pattern  deviates  significantly  from  the  normal 
situation,  will  the  network  have  to  invoke  a  special  handling  mechanism.  In  the  usual 
case,  no  ACK's  are  needed  for  flow  control  purposes  since  our  schemes  use  no  win¬ 
dows.  This  may  save  considerable  processing  capacity  as  well  as  transmission 
bandwidth.  Furthermore,  our  schemes  become  particularly  suitable  if,  for  some  secu¬ 
rity  reason,  the  destination  node  is  not  allowed  to  know  the  identity  of  the  source,  and 
hence  no  ACK  can  be  routed  back. 

(2)  The  second  scheme  can  easily  be  adjusted  to  the  ideal  flow  control  scheme  as  closely  as 
we  desire.  With  most  other  flow  control  schemes,  we  usually  have  no  idea  how  to 
achieve  such  behavior 

(3)  Oui  schemes  are  easy  to  implement.  No  complicated  software  or  algorithms  are 
involved. 

(4)  Most  importantly,  these  procedures  are  easily  evaluated  analytically.  They  can  save 
considerable  effort  in  the  network  development  phase.  One  need  not  write  large, 
cumbersome,  hard-to-debug  programs  to  determine  the  performance  impact  of  these 
flow  control  schemes  on  the  network  performance.  Also,  the  computational  simplicity 
of  our  schemes  have  allowed  us  to  easily  incorporate  many  other  aspects  of  network 
protocols  (such  as  piggy-back  ACK's.  and  time-outs)  into  our  model. 

We  wish  to  comment  on  some  of  our  basic  assumptions.  First  we  have  assumed  that 
we  have  an  infinite  buffer  in  each  node.  This  assumption  liberates  us  from  the  painful  storage 
arrangement  (for  flow  control  purpose)  as  well  as  the  awful  computational  complexity.  As 
storage  elements  become  cheaper  and  cheaper,  our  assumption  becomes  more  and  more  accept¬ 
able.  Second,  we  have  assumed  that  a  node  takes  "no  time"  to  process  messages  so  that  the 
only  delay  is  because  of  transmission.  Though  this  is  quite  true  today,  we  would  like  to  warn 
the  readers  that  this  might  be  reversed  as  faster  and  faster  channels  such  as  digitized  channels, 
and  optical  fibers  [RAWS  78!  introduce  themselves  into  the  world.  Third,  we  have  neglected  to 
model  the  point  to  point  acknowledgement  because  this  results  in  a  simpler  mathemat.zal 
model.  With  better  and  better  channels  for  digital  communication  available  now  and  in  the 
future  we  believe  people  may  simply  abandon  the  point  to  point  acknowledgement  and  let 
errors,  if  any,  be  recovered  end-to-end,  or  by  a  time-out  scheme.  Lastly,  we  wish  to  comment 
that  the  control  scheme  in  this  chapter  aims  to  control  the  average  behavior  of  a  network  only. 
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It  does  not  stop  input  when  occasionally  the  network  is  in  a  congested  status.  Hence  some 
complimentary  methods  such  as  local  congestion  or  dynamic  flow  control  schemes  are  needed 
to  overcome  this  weak  point.  However,  as  pointed  out  before,  this  is  a  common  shortage 
shared  by  almost  all  global  flow  control  schemes.  Some  dynamic  flow  control  schemes  are  dis¬ 
cussed  in  Chapter  5. 


7? 


CHAPTER  5 

DYNAMIC  FLOW  CONTROL  SCHEMES 

There  is  a  Chinese  saying,  "No  flower  can  be  fresh  for  a  hundred  days,  and  no  man 
can  be  sound  for  a  hundred  weeks."  Indeed,  nothing  in  this  world  is  everlasting.  Therefore  no 
long-term  plan  can  be  carried  out  without  dynamically  adapting  itself  to  the  changing  environ¬ 
ment. 


The  flow  control  schemes  discussed  in  Chapter  3  and  4  are  static.  They  work  well  if  the 
throttling  thresholds  (e.g.,  permit-generation  rates)  at  various  entry  points  of  the  network  are 
set  properly  according  to  the  external  demands  (input  rates)  at  these  entry  points  as  well  as  to 
the  network  status. 

Now,  if  we  can  dynamically  adjust  the  throttling  thresholds  according  to  changing  exter¬ 
nal  demands  and  the  internal  state,  we  shall  have  a  high  performance  network. 

Four  problems  arise  immediately.  First,  in  a  geographically  distributed  network,  who  is 
responsible  for  obtaining  the  information  about  the  situation  of  external  demands  and  the  net¬ 
work  status?  Second,  given  that  these  responsible  "administrative  centers”  have  the  informa¬ 
tion,  what  should  they  do?  Third,  how  often  should  we  readjust  the  throttling  thresholds? 
And  fourth,  should  the  adjustment  be  done  globally  or  in  part  of  the  network  at  a  time'’ 

These  are  questions  with  no  simple  answers.  In  fact  there  has  been  a  lack  in  the  litera¬ 
ture  in  solving  these  problems.  Kermani  and  Kleinrock  [KERM  80)  investigated  a  dynamic 
control  scheme  in  which  the  destination  for  a  single  flow  tries  to  control  the  source  input  rate 
according  to  its  own  buffer  utilization.  This  scheme  is  intended  to  match  the  speed  of  the 
source  and  the  destination  of  a  single  flow.  A  better  control  scheme  must  take  the  speed  of  the 
network  and  the  effect  of  side  traffic  into  account  since  a  destination  may  cry  for  more  input 
while  the  network  is  congested.  Also  it  is  worthwhile  to  mention  that  to  analyze  a  single  flow 
scheme  as  simple  as  the  one  that  Kermani  and  Kleinrock  investigated,  they  had  to  use  the 
theory  of  "Markov  decision  processes”,  and  only  a  heuristic  solution  could  be  obtained.  One  is 
then  easily  convinced  that  there  hardly  exists  an  easy  way  to  analyze  dynamic  flow  control  for  a 
large,  geographically  distributed,  multi-variate  network.  With  this  in  mind  one  may  be  less 
reluctant  to  accept  the  dynamic  control  schemes  that  we  propose  and  analyze  later  in  this 
chapter. 


In  this  chapter  we  will  assume  that  although  the  external  demands  do  keep  changing, 
they  do  not  change  too  rapidly  (compared  to  the  relaxation  time  of  the  system*).  This  means 
that  the  rapid  changes  are  considered  as  fluctuations  of  traffic,  and  only  the  D  C.  shifts  are  con¬ 
sidered  as  traffic  changes. 


*  The  relaxation  time  can  he  regarded  as  the  lime  period  for  a  system  to  enter  into  the  new 
steady  state  from  the  old  steady  state  because  of  a  step  change  in  the  input  rate.  See  [KLEI 
76). 
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In  the  schemes  we  discuss  below,  we  always  require  that  the  adjustment  be  done  at  reg¬ 
ular  time  intervals,  each  of  which  is  much  larger  than  the  relaxation  time  of  the  system.  We 
call  these  adjusting  moments,  the  "control  points.” 

S.l  A  General  Dynamic  Control  Scheme 


To  answer  the  four  questions  raised  above,  one  might  suggest  the  following.  Let  some 
node  in  the  network  be  the  central  administrative  node  to  which  every  node  in  the  network 
constantly  reports  the  external  demand  at  its  site.  The  central  node  then  regularly  examines 
the  external  demands  of  the  whole  network,  and  then  tries ’to  allocate  the  network  resources 
(e.g.,  permit  generation  rates)  properly.  By  'properly'  we  mean  to 

( 1 )  satisfy  the  demands  most  fairly; 

(2)  maximize  the  total  throughput  of  the  network;  and 

(3)  limit  the  delay  below  a  certain  acceptable  value. 


One  way  to  do  this  is  to  solve  the 


minimize 


subject  to 

a  fxC,  ^  £  g,  ,  V  physical  channel  / 

rt  R  Id 

and 


following  nonlinear  programming. 


Kr  ^  y,  V  logical  channel  r 

where  y,  is  the  external  input  rate  of  logical  channel  r,  and  g,  is  the  allocated  permit  genera¬ 
tion  rate  for  that  logical  channel;  a  is  a  multiplier  <1  to  keep  utilization  of  channels  <1;  pC, 
is  the  service  rate  at  physical  channel  /;  and 

R  < / )  =  {logical  channel  r  ]  whose  messages  route  through  physical  channel  /) 


The  above  nonlinear  program  tries  to  minimize  the  sum  of  the  squares  of  the  percentage  deficit 


yr~  Hr 

y> 


that  the  network  owes  to  a  logical  channel  r  while  keeping  the  total  throughput  ]£  g, 


maximal.  And  this  is  under  the  condition  that  none  of  the  physical  channels  is  overloaded  and 
that  none  of  the  g,'s  are  over  allocated  (<y,L 


The  advantage  of  a  scheme  such  as  this  one  is  that  the  network  resources  can  be  allo¬ 
cated  most  efficiently  because  the  entire  situation  of  a  network  is  taken  into  account  in  the  allo¬ 
cation  decision  process.  Yet  there  are  two  disadvantages.  First,  because  it  is  centralized,  if  the 
central  node  is  down,  the  flow  control  cannot  function  at  all.  Second,  if  the  network  is  large, 
with  hundreds  of  nodes  and  channels,  then  the  calculation,  transmission  and  processing  of 
updates  is  so  difficult  and  time  consuming  that  the  scheme  may  end  up  with  no  ability  to  con¬ 
trol  the  network  in  real  time.  In  fact,  these  considerations  motivate  us  to  investigate  the  fol¬ 
lowing  two  distributed  dynamic  flow  control  schemes. 
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5.2  Distributed  Dynamic  Flow  Control  —  Scheme  1 

5.2.1  Some  Observations  from  Kleinrock's  Delay  Equation  IKLEI  76l 


We  start  with  the  formula 


T.-  I 


,i  nr )  M 


C,  -  S. 


(5.1) 


where  T,  is  the  average  round  trip  delay  of  logical  channel  r:  S,  is  the  sum  of  the  traffic  over 
logical  channels  whose  messages  are  routed  through  physical  channel  /;  and  l(r)  =  (physical 
channel  /  I  physical  channel  t  belongs  to  the  path  for  logical  channel  r\ 


Consider  the  simplest  (but  non-realistic)  case  that  the  logical  channel  consists  of  only 
one  physical  channel  i*.  Then 


T, 


(5.2) 


i.e.. 


5,-  =  mO 


Tr 


(5.3) 


Let  p*  <  1  be  the  maximal  allowed  utilization  factor  for  channel  /*  (for  a  finite  maxi¬ 
mal  average  delay).  Then  corresponding  to  p*,  we  have  the  maximum  traffic  (through  channel 
/) 


s*  =  pVO 

(5.4) 

and  the  maximum  channel  delay 

r*  -  1 

(5.5) 

mO-S* 

i.e.. 

S*  =  pC>--^ 

(5.6) 

From  Eqs.  (5.3)  and  (5.6)  we  have 

8S,. 

i 

-|| 

ii 

Co 

1 

• 

Co 

*111 

± 

(5.7) 

Tr 

T* 

Equation  (5.7)  tells  us  that  in 

the  one  hop  case. 

by  measuring  T, 

.  if  T,  <  T*  ,  one 

can  increase  the  input  rate  by 

I  I 

Tr  T' 

without  exceeding  the  maximum  channel  delay  T* 


This  observation  is  the  cornerstone  of  the  dynamic  distributed  flow  control  scheme  that 
we  investigate  in  this  section. 
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Usually  the  path  of  a  logical  channel  consists  of  K  ( A"  >  1 )  physical  channels.  Imagine 
that  we  sit  at  the  source  of  logical  channel  r.  We  measure  the  average  round  trip  delay  T, 
there.  We  assume  that  Tr  is  mainly  because  of  the  delay  at  some  single  physical  channel  i* 
along  the  path  of  logical  channel  r  +  .  Then  if  T,  <  T,0  <  Trg  is  the  maximal  allowed  round  trip 
delay  of  logical  channel  r),  we  will  increase  5,  by  85,  such  that 

85,  =  -r-jr-  (5.8) 

'r  'ro 

Let  us  illustrate  the  effect  of  doing  this  in  Fig.  5.1. 


Figure  5.1.  Multi-step  adjustment  of  input  rate. 

Assume  K  =  3  (again,  "3”  is  chosen  just  for  simplicity;  it  may  be  unrealistic  since 
round-trip  paths  are  more  likely  to  consist  of  an  even  number  of  hops).  And  for  the  sake  of 
easy  illustration  let  us  assume  that  the  channel  utilizations  and  the  channel  speeds  in  these 
three  channels  are  the  same  so  that  the  actual  delay  in  each  channel  T,  is 

t  -  r,/3 

Let  Tr  be  the  measured  average  round-trip  delay  at  the  /"'  control  point.  Now  since  at 
the  source  we  think  Tr]  (=  3  see  Fig.  5.1)  is  the  delay  at  channel  /*,  we  also  would  think 
that  the  traffic  at  channel  i*  is  5,(  while  it  is  actually  5,. .  Thus,  we  conclude  that  we  can 
increase  5,  by 


+  This  is  not  an  unrealistic  assumption.  The  channel  capacities  and  the  traffic  matrix  in  a 
network  are  usually  heterogeneous.  Therefore,  some  channel  may  be  saturated  before  other 
channels  are. 
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y 


rt'\  T  T 

''ll 

which  brings  .S'  •  10  S,v  At  the  second  adjusting  moment,  the  measured  delay  would  become 
r.,(  =  3  T This  w  ill  give  us  ft  S, ,.  By  adding  ft.S'(l  to  S,  we  will  have  .S  ■  =  S..  And  in  turn, 
at  the  third  adiusting  moment  we  will  obtain  the  measured  delay  7^1=3  7". .  Eventually  we 
will  reach  7, 

Based  on  the  above  observation,  the  following  distributed  dynamic  control  scheme  is 
proposed  We  again  emphasize  that  the  assumption  that  delay  is  lumped  at  a  single  channel  is 
not  a  bad  one  since  the  network  capacities  are  usually  heterogeneous  and  the  traffic  uneven 

5.2.2  The  Measure-and-Control  Scheme 

In  this  section  we  present  a  scheme  that  controls  the  flow  by  having  each  source  meas¬ 
ure  the  round-trip  delay  of  its  message,  and  according  to  this  measurement,  set  an  appropriate 
permit  generation  rate  (see  Chapter  4)  for  itself  unless  it  obtains  a  control  signal  telling  it  not 
to  do  so  We  describe  the  scheme  in  detail  by  presenting  the  following  rules: 

(It  Originally  each  logical  channel  /  is  assigned  a  basic  legal  permit-generation  rate  ,i», 
This  is  the  rale  at  which  a  source  can  operate  whenever  it  desires  (e.g..  as  the  input  rate 

exceeds  v,  l  In  the  long  run.  as  long  as  each  source  operates  at  this  rate,  the  network 

is  not  going  to  be  overloaded 

<2>  Each  source  measures  the  external  input  rate  y,.  and  the  average  round  trip  delay  T, 

'.'I  At  each  control  point  (i  e.  adiusting  moment!  if  a  source  finds  that  y,>.u,  (.e,  is  the 

current  permit-generation  rate'  and  7,  <  T,n.  it  increases  v.  by 

I  I 

ne.  =  j  j 

But  if  7,  >  7,  .  it  does  nothing 


If  n  finds 


then  it  reduces  e,  to 


Whenever  a  physical  channel  (not  necessarily  at  control  points)  finds  its  backlog  is 
larger  than  \„.  it  issues  an  instruction  to  all  the  sources  whose  messages  are  routed 
through  it  T  his  instruction  tells  each  source  to  reset  its  e,  to  g,  (if  v,  >.«*,  >  until 
further  notification 

A  channel,  when  its  backlog  is  reduced  to  A ,  from  Y„.  will  issue  a  relief  signal  to  the 
relevant  sources  so  that  they  can  start  the  measure-and-control  procedure  again. 

C  ontrol  instructions  are  sent  with  the  highest  priority  ( i.e  .  they  bypass  queues  for  data 
messages)  The  reason  for  points  <51.  (M  and  (7)  is  that  when  T,  is  large  (i  e..  some¬ 
where  the  backlog  is  laige).  the  source  can  no  longer  control  the  traffic  in  real  time 
because  it  receives  the  information  too  late  The  channel  with  the  critical  backlog  has 
to  immediately  instruct  the  source  to  reduce  its  input  rale  by  cutting  its  permit 


generation  rate  to  a  safe  value. 

This  scheme  is  certainly  distributed  and  dynamic.  Failures  in  some  parts  of  a  network  do  not 
affect  (in  the  flow  control  sense)  the  other  irrelevant  parts  of  the  network.  No  complicated 
computation  is  involved  and  information  is  not  transferred  back  and  forth  globally,  but  only 
among  relevant  parties.  Fig.  5.2  shows  a  typical  performance  curve  of  two  logical  channels 
sharing  some  common  path  that  we  may  see  when  we  apply  the  scheme  to  a  network. 


DELAY  OF  LOGICAL 
CHANNEL  1 


RATE 

Figure  5.2.  Topical  dela>  cune  of  Scheme  I. 


In  this  diagram,  at  a  control  point  r,,  the  measured  input  rate  of  logical  channel 
A  (A  =  l,2)  is  ykl.  Between  (r |,  r2 ),  the  input  rates  are  small  ( < < j?,0) .  so  that  the  measured 
deity  7j  of  logical  channel  1  is  also  small  and  g,=y,  (rule  4).  At  r2,  the  measured  input  rate  of 
logical  channel  1  steps  to  yt2  (>tfio)  and  it  first  steps  its  g-value  to  £|0  (rule  1).  Then  logical 
channel  1  increases  its  jc-value  according,  to  rule  3  (but  if  the  resultant  g  value  is  smaller  than 
Kio,  it  is  set  to  g  —  atio-  according  to  rule  ]).  It  would  take  several  control  points  for  logical 
channel  1  to  increase  its  /?-value  to  the  final  value  (corresponding  to  Tx0).  But  before  this 
final  value  is  reached,  the  measured  input  rate  of  logical  channel  2  increases  to  y23  at  ih  and 
the  measured  delay  T2  is  near  T20.  Logical  channel  2  would  just  operate  at  g20  according  to  rule 
1.  This  will  cause  some  physical  channel  to  be  overloaded  (p,-  >  1),  and  the  delay  increases 
linearly  until  /*(  where  the  backlog  in  physical  channel  i*  reaches  the  limit  Nu.  The  g-values  of 
logical  channel  1  and  2  are  then  reset  to  gi0  and  g20  respectively  (rule  5).  This  causes  the  back¬ 
log  to  decay.  At  i'2  the  relief  threshold  NL  is  reached,  and  the  measure-and-control 
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mechanism  starts  again  (rule  6).  The  same  cycle  repeats  during  r2*,/4*).  At  t4  the  input  rate 
of  logical  channel  1  steps  down,  while  that  of  logical  channel  2  is  still  high.  Logical  channel  2 
starts  to  gain  its  jc-value  at  each  control  point.  The  delay  of  logical  channel  1  still  increases 
since  they  share  some  physical  channels  in  their  paths.  At  r4  the  input  rate  of  logical  channel  2 
also  steps  down.  Therefore  the  delay  also  decays  to  a  lower  value. 

5.3  The  Simulation  Results 


We  tested  the  measure-and-control  scheme  by  simulation  on  our  little  network  in  Fig. 
4.2.  Here  we  change  the  capacity  of  physic.:!  channel  5-3  to  48  (messages  per  second)  in  order 
to  make  the  network  a  little  bit  non-uniform.  Other  physical  channels  have  capacities  of  50. 
The  results  are  shown  in  Figs.  5.3(a),  (b),  (c)  and  (d). 


In  the  simulation  we  kept  the  input  rates  of  all  logical  channels  (other  than  5-4  and  5- 
2)  randomly  under  3  messages  per  second,  in  order  to  single  out  the  effects  of  changing  the 
input  rate  of  logical  channels  5-2  and  5-4.  The  Kro  values  are  set  at  5.7  so  that  when  all  logical 
channels  operate  at  this  value  the  most  loaded  physical  channel  has  traffic  at  5.7  x  8  =  45.6 
messages  per  second  (8  comes  from  the  fact  that,  for  example,  through  physical  channel  3-2  we 
have  four  forward  logical  channels  5-2,  5-4,  3-2,  3-4  and  four  acknowledgement  channels  for 


logical  channels  4-3,  4-5,  2-3,  2-5). 


The  maximum  utilization  factor  p* 


45.6 

48 


0.95  (for 


physical  channel  5-3). 


The  input  rates  of  logical  channels  5-2  and  5-4  are  shown  in  Fig.  5.3(a).  The  measured 
delays  are  shown  in  Fig.  5.3(b).  The  control  point  intervals  are  one  minute  long  7.  Observe 
the  similarity  between  Fig.  5.3(b)  and  Fig.  5.2.  The  reset  instants  are  indicated  by  the  vertical 
lines.  The  upper  limit  N„  on  the  backlog  of  any  physical  channel  is  60,  and  the  relief  threshold 
is  30.  The  path  of  logical  channel  5-4  is  longer  than  that  of  5-2.  Hence  the  delay  of  5-4  is 
longer  than  that  of  5-2.  However,  at  the  peaks  the  two  delay  curves  come  very  close  to  each 
other  This  is  because  one  of  the  physical  channels  is  overloaded  while  others  still  have  a  utili¬ 
zation  factor  much  smaller  than  one.  This  justifies  our  assumption  of  the  existence  of  a  critical 
physical  channel. 


In  Fig.  5.3(c)  we  show  the  adjustments  of  the  permit  generation  rates  of  the  two  logical 
channels.  The  vertical  lines  again  indicate  the  resetting  moments.  Observe  that  logical  channel 
5-2  always  gains  in  its  .e-value  faster  than  logical  channel  5-4.  The  reason  behind  this  is 
explained  below. 


Let  r,  and  r2  be  two  logical  channels  such  that  n,;  contains  n,r  where  nr,  '=1.2,  is 
the  path  of  logical  channel  r,.  Suppose  that  at  the  same  control  point  the  measured  average 
delays  are  Tr>.  and  T,:  respectively,  and  they  are  somewhat  smaller  than  T,\0  and  Trio  (the  max¬ 
imum  allowable  values)  respectively. 

Then. 


+  If  p*  =  0.95  for  our  little  network  then  the  relaxation  time  for  the  queue  at  physical  channel 


I  IS  T  ■ 


45. 6(  1/48) 2 


=  7seconds.  1/48  seconds  is  the  service  time  of  physical 


(p*-!)2  (0.05) 2 

channel  5-3.  This  shows  that  one  minute  is  much  larger  than  the  relaxation  time. 
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DELAY  OF  LOGICAL  CHANNELS  5-4  AND  5-2  ISECONDS) 


INPUT 

RATE  (SEC"1) 


Figure  5.3(a).  The  input  rates  of  logical  channels  5-2  and  5-4. 


Figure  5.3(b).  The  simulated  delav  of  Scheme  I. 
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SIMULATION  TIME  (SECS) 


Figure  5.3(c).  The  simulated  permit  gain  of  Scheme  i 


i - 1 - 1 - 1 - 1 - 1 - 1 - r 


Figure  5.3(d).  The  simulated  throughput  of  Scheme  I 
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i.e.,  fi.i v2  <  fi^,r  We  have  used  ihe  fact  that 


a  +  b  +  c  a  +  b 
\_  _  1 
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<  0 


TrU 


Tr>0 


—  0. 


This  is  exactly  what  we  observed  in  Fig.  5.3  since  7r5_4  contains  ns-2- 

Intuitively  speaking,  this  is  because  a  logical  channel  with  a  longer  path  tends  to  be 
more  inaccurate  than  one  with  a  shorter  path  when  it  assumes  that  the  delay  is  lumped  at  some 
single  critical  physical  channel,  while  actually  every  physical  channel  in  the  path  contributes  to 
the  delay. 

One  may  wonder  if,  in  order  to  eliminate  the  above  mentioned  "capture"  effect,  we 
should  take  the  path  length  into  consideration,  e  g.,  to  multiply  the  incremental  g-value  with  a 
parameter  <»,  ^  1,  that  is  a  function  of  path  length  of  the  logical  channel  r.  Indeed,  we  can  do 
so,  except  for  the  fact  that  it  is  easier  to  make  a  misjudgement  for  the  incremental  £-value 
since  ur  is  also  a  function  of  the  traffic  matrix,  which  may  change  dramatically  with  time. 
Thus,  there  is  no  simple  optimal  way  to  specify  the  value  of  ar. 

The  difficulty  in  accurately  adjusting  the  , e-value  is  an  inherent  nature  of  distributed 
decision  processes. 

Fig.  5.3(d)  shows  the  throughput  of  logical  channels  5-2  and  5-4.  One  should  compare 
this  with  the  input  rates.  The  scheme  does  dynamically  control  the  flow  reasonably  well  (except 
for  the  capture  effect  indicated  before). 
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The  oscillation  can  be  minimized  by  a  smoother  resetting  mechanism.  For  example,  a 
scheme  with  multi-threshold  levels  and  multi-steps  for  resetting  g-values  can  be  used.  Indeed, 
we  shall  use  this  type  of  resetting  scheme  in  an  example  of  scheme  2. 

5.4  Distributed  Dynamic  Flow  Control  -  Scheme  2 

In  this  section  we  present  a  less  distributed  dynamic  control  scheme  that  allocates  the 
g-values  more  effectively. 

In  this  scheme,  at  r  seconds  prior  to  each  control  point,  the  sources  and  physical  chan¬ 
nels  exchange  information  for  Nc  cycles.  Ne  is  a  proper  number  chosen  by  the  designer  of  the 
control  scheme.  At  each  cycle  a  source  passes  information  about  the  current  demand  and  fig- 
value  gained  from  the  last  cycle  to  the  physical  channels  along  the  path  of  the  logical  channel, 
and  the  physical  channels  pass  the  information  about  their  allocation  of  current  fig-values  to  the 
source.  The  details  are  described  below. 

5.4.1  The  Overall  Rules 

(1)  t  seconds  prior  to  each  control  point,  execute  the  "g-value  allocation"  mechanism  in 
Sec.  5.4.2  for  Nc  cycles. 

(2)  At  each  control  point,  each  source  adjusts  its  g-value  to  that  obtained  by  rule  (1),  and 
the  network  operates  at  these  new  g-values. 

(3)  Whenever  the  input  rate  yr  of  a  source  r  steps  up  from  a  low  value  (<gr0)  to  a  high 
one  (>g,0),  it  can  operate  at  g,0  immediately  whether  or  not  the  next  control  point  has 
arrived. 

(4)  If  rule  (3)  ever  causes  any  physical  channel  to  be  overloaded,  the  physical  channel  will 
activate  a  resetting  mechanism  as  various  thresholds  of  backlog  are  reached. 

5.4.2  g  Value  Allocation  Rules  at  Each  Cycle 

(1)  Each  source  /  passes  (y, _8g,)  to  each  physical  channel  /  along  its  path  to  its  destination, 
where  y,  is  the  source’s  current  demand  and  fig,  is  the  incremental  g-value  gained  dur¬ 
ing  the  last  cycle.  Note  that  for  the  first  cycle,  y,  is  the  external  input  rate  and 
fig,  -  0.  The  demand  of  the  next  cycle  of  source  /  is  y,  =  y,  of  the  current  cycle  less 
fig,  gained  in  this  cycle. 

(2)  A  physical  channel  f,  upon  receiving  (y,,  fig,)  V  source  ./  6  R  (/),  allocates  its  residual 
current  capacity  C,  to  these  sources  such  that  source ./  receives  capacity  A„  according  to 
the  allocation  algorithm  in  Sec.  5.4.3.  Again,  note  that  C,  at  the  first  cycle  is  the  capa¬ 
city  of  channel  /.  C,  at  the  current  cycle  is 

C,  =  (C,  of  Iasi  cycle)—  £  fig,  (Iasi  cycle) 

/€ 

(3)  Channel  /  then  passes  4„  to  e,.ch  source  /'.  Source  /  picks  up  its  new  fig,  value  with 

fig,  -  min  A  fig,  is  then  added  to  the  g-value  allocated  to  source  <  so  far  (up  to  the 

«</(/) 

last  cycle). 


Table  5.1  Illustrative  example  of  A„  allocation  algorithm. 


Initial 

1st 

Iteration 

2nd 

Iteration 

3rd 

Iteration 

Final 

Values 

Demand 

5(» 

0  s 

Oo 

l_ 

0o 

0s 

Subscript 

7o 

2s 

o2 

00 

07 

e  Allocated 

% 

4> 

02 

09 

12«, 

7s 

52 

32 

3, 

Capacity 

Remaining 

30 

10 

_ 4 _ 

0 

0 

Table  5.1  shows  the  result  of  a  capacity  allocation  example  where  a  physical  channel 
tries  to  allocate  its  capacity  to  four  logical  channels.  In  this  example,  the  four  sources  route 
their  messages  through  a  physical  channel  with  a  capacity  of  30  The  initial  demands  of  these 
logical  channels  are  5,  7,  9  and  12,  respectively.  The  initial  allocated  .e-values  are  0.  After  the 
first  iteration  (see  the  allocation  flow  chart)  the  allocated  , e-values  are  5. 5. 5, 5.  The  residual 
capacity  of  the  physical  channel  is  10.  and  the  current  demands  become  0,2.4. 7.  After  the 
second  iteration  the  allocated  incremental  .e-values  are  0.2. 2, 7,  the  remaining  capacity  is  4,  and 
the  current  demands  are  0,0, 2, 5.  After  the  third  iteration,  the  incremented  e-values  are 
0,0, 2, 2.  the  residual  capacity  is  0,  and  the  current  demands  are  0,0, 0.3.  So.  finally  the  four 
sources  have  e-values  5. 7. 9, 9  with  the  fourth  source  not  satisfied  by  3  units.  The  capacity  of 
the  physical  channel  is  used  up. 
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Figure  5.4.  Illustrative  network  for  dynamic  control  scheme  2. 


To  illustrate  how  the  flow  control  scheme  works,  we  apply  3  cycles  of  the  set  of  rules  in 
Sec.  5.4.2  to  the  network  shown  in  Fig.  5.4.  Here  we  show  a  network  with  six  physical  chan¬ 
nels  having  capacities  as  denoted,  and  we  show  four  logical  channels  L},L2,L2,  and  L4  with  ini¬ 
tial  demands  of  25,  20,  15,  and  25.  The  results  of  applying  3  cycles  of  the  tf-value  allocation 
rules  are  summarized  in  Table  5.2. 

Table  5.2  Results  of  applying  3  cycles  of  #-value  allocation  rules. 
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Let  us  follow  the  activity  of  physical  channel  2.  Initially  it  has  a  capacity  of  50.  At  the 
first  cycle  it  allocates  20  and  25  to  Z.,  and  L2  respectively.  Yet  L\  and  L2  obtain  only  15  and 
15  of  the  .e-values  because  the  allocation  made  by  physical  channel  1  is  15,  15,  15,  for 
L\,L}.Li  At  the  beginning  of  the  second  cycle,  physical  channel  2  has  capacity  of  20  (  =  50- 
15-15)  remaining.  The  residual  demands  of  L\  and  L2  are  10  and  5  so  it  allocates  10.5  to  L\ 
and  L 2  respectively.  But  since  channel  1  allocates  5/3.  5/3,  S/3  to  L\,  /.2,  Ly  and  channel  3 
allocates  0  and  0  to  l.\  and  L4,  the  6,t,*-values  obtained  by  L\  and  L2  in  this  cycle  are  0.  5/3. 
Now,  at  the  beginning  of  the  third  cycle  channel  2  has  a  capacity  of  55/3  (  =  20-5/3)  remaining. 
It  in  turn  allocates  10  and  10/3  to  /.,  and  1.2  But  fit?  gained  by  L\  and  L2  in  this  cycle  is  0. 
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It  in  turn  allocates  10  and  10/3  to  L\  and  But  8 g  gained  by  L\  and  L}  in  this  cycle  is  0, 
10/9.  Therefore,  finally  channel  2  has  capacity  of  155/9  (—  55/3  -  10/9)  remaining.  The 
activities  of  other  physical  channels  can  be  followed  similarly. 

The  resulting  ^-values  obtained  by  Z.),Z.2,Z.3  and  £4  are  15,  17.78,  10,  and  15. 

The  final  ^-values  are  the  combined  products  of  several  factors,  namely,  the  fairness 
consideration,  the  topology  constraint,  the  channel  capacities  and  the  routing  algorithm. 
Observe  that  as  Nc  — ' *  <»,  g-values  converge  to  their  asymptotic  values. 

Since  rule  (3)  in  Sec.  5.4.1  allows  a  source  to  step  up  to  its  legal  g,0-value,  overload 
occasionally  occurs.  Therefore  a  resetting  mechanism  as  slated  in  rule  (4)  is  necessary. 

One  example  of  a  resetting  mechanism  is  shown  in  Fig.  5.5.  In  the  figure,  the  horizon¬ 
tal  lines  represent  the  backlog  of  the  physical  channel.  If  every  physical  channel  is  in  region 
(a,b),  (b,d),  (e,f),  or  (h,i)  the  g-values  are  decided  at  each  control  point  by  the  rules  in  Sec. 
5.4.1.  If  a  physical  channel  is  on  (b,c)  or  (c,e),  the  ^-values  of  all  sources  whose  messages 
pass  through  it  are  instructed  to  reduce  its  g- value  by  10%.  But,  if  source  r  has  a  g-value  gr 
smaller  than  grQ,  'l  would  still  operate  at  the  g,  level  even  if  it  is  instructed  to  reduce  10%. 
Finally,  if  it  is  on  (c,g)  or  (g,h),  the  relevant  sources  can  only  operate  at  grQ%  (or  lower)  to 
force  the  network  to  be  stabilized. 


Figure  5.5.  A  resetting  mechanism. 


5.5  The  Simulation  Results 

We  tested  scheme  2  on  the  little  network  in  Fig.  4.2  that  we  have  mentioned  so  many 
times.  We  use  the  same  parameter  values  as  in  Sec.  5.3. 
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the  .e-values  of  5-4  and  5-2  allocated  at  control  points.  Table  5.3  shows  the  resetting  activity 
during  the  simulation.  Figs.  5.6(c)  and  5.6(d)  show  the  throughputs  and  delays  of  the  logical 
channels  5-4  and  5-2. 


Table  5.3.  Reset  activity. 
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61 

1706.956 
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61 

One  observes  that  indeed  there  is  no  more  capture  effect  in  this  scheme,  e-values  are 
assigned  according  to  the  demand  and  the  network  capacity.  The  reader  is  urged  to  compare 
the  input  rates  and  the  e-values  assigned.  Also  note  that  it  takes  only  one  control  point  to  allo¬ 
cate  the  e-values,  while  in  the  first  scheme  it  takes  several  control  points.  Yet  the  effectiveness 
of  allocating  capacity  is  gained  at  the  cost  of  passing  information  back  and  forth  several  rounds 
between  the  relevant  parties.  (In  our  simulated  case  the  cost  is  1.5  sec.  per  control  point.) 

Refer  to  Table  5.2.  It  is  observed  that  if  in  a  cycle,  a  physical  channel  /  receives  a 
( y ,,  6.e,)  from  source  /  with  6.e,  =0.  then  the  physical  channel  can  actually  determine  that 
some  other  physical  channel  assigned  a  zero  .e-value  to  source  /  during  the  last  cycle,  and  that 
physical  channel  is  to  assign  a  zero  e-value  to  source  /  again  in  this  cycle.  This  means  that 
physical  channel  /  can  regard  the  residual  demand  y,  of  source  /  to  be  zero  even  though  it  is 
not.  By  doing  this,  channel  /  can  allocate  more  of  its  residual  capacity  to  other  sources  (whose 
messages  pass  through  it)  with  fix,  ^  0  and  y,  ^  0.  This  results  in  a  more  effective  allocation 
mechanism. 

5.6  Conclusion  and  Remarks 

People  have  long  recognized  that  dynamic  flow  control  is  needed  to  accommodate  a 
changing  environment.  Yet  the  difficulty  lies  in  the  fact  that  the  geographic  distribution  of  the 
system  makes  it  difficult  to  obtain  the  information  needed  in  time  for  a  real-time  dynamic  con¬ 
trol.  Also,  the  largeness  of  the  system  may  make  the  computational  effort  too  much  to  be  prac¬ 
tical  Furthermore,  it  is  also  realized  that  in  a  distributed  system,  we  should  not  rely  on  one  or 
a  few  centers  to  make  decisions  that  are  important  for  the  whole  system.  Otherwise  the  whole 
system  may  be  seriously  affected  when  these  centers  are  down. 


THROUGHPUT  (PKTS/SEC1  DELAY  OF  LOGICAL  CHANNELS  5-4  and  5-2  ISECSI 


Figure  5.6(c).  The  delay  of  Scheme  2. 


Figure  5.6(d).  The  throughput  of  Scheme  2. 
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The  above  thoughts  then  motivate  our  research  in  distributed  dynamic  control.  What 
we  really  want  is  to  seek  methods  whereby  each  individual  party  of  the  system  can  act  properly 
by  consulting  as  few  other  parties  of  the  system  as  possible. 

We  immediately  see  the  contradiction  here.  In  a  large  dynamically  changing  system  if 
you  want  proper  behavior  you  must  keep  track  of  the  slate  of  the  system  and  that  requires 
some  consulting  between  parties. 

In  the  first  scheme  (Sec.  5.2)  the  "consulting"  is  imbedded  in  the  measurement  of  the 
round  trip  delay,  that  tells  the  source  about  the  collective  activity  level  of  the  physical  channels 
along  its  path.  The  only  one  who  makes  a  decision  for  the  source  is  the  source  itself.  (Of 
course,  when  sources  make  mistakes  we  need  someone  to  correct  them.  We  let  the  critical 
channel  do  this.) 

In  the  second  scheme  (Sec.  5.4)  both  sources  and  physical  channels  consult  with  each 
other  to  make  decisions.  One  thing  we  see  is  that  the  first  scheme  involves  fewer  parties  and 
less  information  than  the  second  scheme  in  making  decisions.  Yet  the  second  produces  better 
decisions  than  the  first  scheme. 

The  above  discussion  seems  to  persuade  us  to  believe  that  there  exists  an  uncertainty 
principle  in  the  area  of  distributed  dynamic  flow  control,  that  may  roughly  be  stated  as  follows. 

Let  /  A  degree  of  independence  of  an  individual  part  to  make  a  decision  and 
let  £  A  effectiveness  of  the  decision.  Then  for  a  given  network,  we  conjecture 
that  /x£  =  c  where  £,  the  effectiveness  of  the  decision,  is  compared  to  the 
optimal  decision  based  on  some  measure  set  by  the  designer  of  a  system  (for 
example,  the  measure  may  be  the  nonlinear  equation  in  Sec.  5.1);  and  /,  the 
degree  of  independence  in  making  the  decision  may  be  defined,  for  example,  as 
/  =*  (amount  of  information  needed)-1;  and  C,  the  constant,  may  be  a  func¬ 
tion  of  network  topology,  routing,  size,  etc. 

This  is  an  interesting  and  important  research  area.  Unfortunately,  we  have  not  yet  car¬ 
ried  out  that  investigation.  The  interested  reader  is  urged  to  pursue  it. 
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